Nextcloud replication with redis cluster

[Janl1]

Is there a way to use this chart with a redis cluster and turn up the replication of Nextcloud? I am using the Bitnami redis cluster and if I increase the Nextcloud replica count my session is getting killed and i can't log in. If i reduce the replica count back to one everything works as it should.

Using the non redis cluster setup with more than one Nextcloud instance works as well, but not both. My desired state is to ensure high availability. I am using a shared ReadWriteMany pvc, overwriteprotocol is set to https.

[amizzo87]

Changing this line to true in values.yaml should (I think?) work out-of-the-box, but my setup is to set redis.config.php: false and use this config:

redis.config.php: |-
      <?php
      $CONFIG = array (
        'memcache.distributed' => '\OC\Memcache\Redis',
        'memcache.locking' => '\OC\Memcache\Redis',
        'redis' => array(
          'host' => getenv('REDIS_HOST'),
          'port' => getenv('REDIS_HOST_PORT') ?: 6379,
          'password' => getenv('REDIS_HOST_PASSWORD')
        )
      );

[Janl1]

Thank you for your input but i guess you are not getting my point. The default deployment is working fine but I need the redis cluster to ensure high availability,

In my opinion it doesn't make sense that a single nextcloud instance combined with redis cluster works fine but multiple instances doesn't.

[amizzo87]

Ah I see what you mean. Interesting, I don't have that problem with the provided redis cluster setup + multiple replicas. I did, however, have to deploy and install with just one instance, then afterwards ramp up to multiple. But I'm not sure that helps...

[Janl1]

Even if I start with only one instance and add more later the secound instance breaks my session and the nextcloud deployment isn't useable any more. @amizzo87 is there any chance of sharing your current chart?

[amizzo87]

Sure thing, here’s my values.yaml. This works with the current version of the helm chart pulled from this repo:

## Official nextcloud image version
## ref: https://hub.docker.com/r/library/nextcloud/tags/
##
image:
  repository: nextcloud
  tag: 23.0.0-apache
  pullPolicy: IfNotPresent
  # pullSecrets:
  #   - myRegistrKeySecretName

nameOverride: ""
fullnameOverride: ""
podAnnotations: {}
deploymentAnnotations: {}

# Number of replicas to be deployed
replicaCount: 3

## Allowing use of ingress controllers
## ref: https://kubernetes.io/docs/concepts/services-networking/ingress/
##
ingress:
  enabled: false
  # className: nginx
  annotations: {}
  #  nginx.ingress.kubernetes.io/proxy-body-size: 4G
  #  kubernetes.io/tls-acme: "true"
  #  cert-manager.io/cluster-issuer: letsencrypt-prod
  #  nginx.ingress.kubernetes.io/server-snippet: |-
  #    server_tokens off;
  #    proxy_hide_header X-Powered-By;

  #    rewrite ^/.well-known/webfinger /public.php?service=webfinger last;
  #    rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
  #    rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
  #    location = /.well-known/carddav {
  #      return 301 $scheme://$host/remote.php/dav;
  #    }
  #    location = /.well-known/caldav {
  #      return 301 $scheme://$host/remote.php/dav;
  #    }
  #    location = /robots.txt {
  #      allow all;
  #      log_not_found off;
  #      access_log off;
  #    }
  #    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
  #      deny all;
  #    }
  #    location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
  #      deny all;
  #    }
  # tls:
  #   - secretName: nextcloud-tls
  #     hosts:
  #       - nextcloud.kube.home
  labels: {}
  path: /
  pathType: Prefix

host: __baseUrl__
password: __admin-password__
username: admin

# Allow configuration of lifecycle hooks
# ref: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
lifecycle: {}
  # postStartCommand: []
  # preStopCommand: []

nextcloud:
  host: __baseUrl__
  username: admin
  password: __admin-password__
  ## Use an existing secret
  existingSecret:
    enabled: false
    # secretName: nameofsecret
    # usernameKey: username
    # passwordKey: password
    # smtpUsernameKey: smtp_username
    # smtpPasswordKey: smtp_password
  update: 0
  # If web server is not binding default port, you can define it
  # containerPort: 8080
  datadir: /var/www/html/data
  persistence:
    subPath:
  mail:
    enabled: true
    fromAddress: noreply
    domain: __baseUrl__
    smtp:
      host: smtp.sendgrid.net
      secure: ssl
      port: 465
      authtype: LOGIN
      name: apikey
      password: __(sendgrid-apikey)__
  # PHP Configuration files
  # Will be injected in /usr/local/etc/php/conf.d for apache image and in /usr/local/etc/php-fpm.d when nginx.enabled: true
  phpConfigs: {}
  # Default config files
  # IMPORTANT: Will be used only if you put extra configs, otherwise default will come from nextcloud itself
  # Default confgurations can be found here: https://github.com/nextcloud/docker/tree/master/16.0/apache/config
  defaultConfigs:
    # To protect /var/www/html/config
    .htaccess: true
    # Redis default configuration
    redis.config.php: false
    # Apache configuration for rewrite urls
    apache-pretty-urls.config.php: true
    # Define APCu as local cache
    apcu.config.php: true
    # Apps directory configs
    apps.config.php: true
    # Used for auto configure database
    autoconfig.php: false
    # SMTP default configuration
    smtp.config.php: true
  # Extra config files created in /var/www/html/config/
  # ref: https://docs.nextcloud.com/server/15/admin_manual/configuration_server/config_sample_php_parameters.html#multiple-config-php-file
  configs:
    custom.config.php: |-
      <?php
      $CONFIG = array (
        'overwriteprotocol' => 'https',
        'overwrite.cli.url' => '__baseUrl__',
        'filelocking.enabled' => 'true',
        'loglevel' => '2',
        'enable_previews' => true,
        'trusted_domains' => array(
          0 => 'localhost',
          1 => '__baseUrl__'
          )
      );

    redis.config.php: |-
      <?php
      $CONFIG = array (
        'memcache.distributed' => '\OC\Memcache\Redis',
        'memcache.locking' => '\OC\Memcache\Redis',
        'redis' => array(
          'host' => getenv('REDIS_HOST'),
          'port' => getenv('REDIS_HOST_PORT') ?: 6379,
          'password' => getenv('REDIS_HOST_PASSWORD')
        )
      );

    s3.config.php: |-
      <?php
      $CONFIG = array (
        'objectstore' => array(
          'class' => '\\OC\\Files\\ObjectStore\\S3',
          'arguments' => array(
            'bucket'     => '__nextcloudBucket__',
            'autocreate' => false,
            'key'        => '__wasabi-key__',
            'secret'     => '__wasabi-secret__',
            'region'     => 'us-west-1',
            'hostname'   => 's3.us-west-1.wasabisys.com',
            'use_ssl'    => true
          )
        )
      );    

  # For example, to use S3 as primary storage
  # ref: https://docs.nextcloud.com/server/13/admin_manual/configuration_files/primary_storage.html#simple-storage-service-s3
  #
  #  configs:
  #    s3.config.php: |-
  #      <?php
  #      $CONFIG = array (
  #        'objectstore' => array(
  #          'class' => '\\OC\\Files\\ObjectStore\\S3',
  #          'arguments' => array(
  #            'bucket'     => 'my-bucket',
  #            'autocreate' => true,
  #            'key'        => 'xxx',
  #            'secret'     => 'xxx',
  #            'region'     => 'us-east-1',
  #            'use_ssl'    => true
  #          )
  #        )
  #      );

  ## Strategy used to replace old pods
  ## IMPORTANT: use with care, it is suggested to leave as that for upgrade purposes
  ## ref: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
  strategy:
    type: Recreate
    # type: RollingUpdate
    # rollingUpdate:
    #   maxSurge: 1
    #   maxUnavailable: 0

  ##
  ## Extra environment variables
  extraEnv:
  #  - name: SOME_SECRET_ENV
  #    valueFrom:
  #      secretKeyRef:
  #        name: nextcloud
  #        key: secret_key

  # Extra mounts for the pods. Example shown is for connecting a legacy NFS volume
  # to NextCloud pods in Kubernetes. This can then be configured in External Storage
  extraVolumes:
  #  - name: nfs
  #    nfs:
  #      server: "10.0.0.1"
  #      path: "/nextcloud_data"
  #      readOnly: false
  extraVolumeMounts:
  #  - name: nfs
  #    mountPath: "/legacy_data"

  # Extra secuurityContext parameters. For example you may need to define runAsNonRoot directive
  # extraSecurityContext:
  #   runAsUser: "33"
  #   runAsGroup: "33"
  #   runAsNonRoot: true
  #   readOnlyRootFilesystem: true

nginx:
  ## You need to set an fpm version of the image for nextcloud if you want to use nginx!
  enabled: false
  image:
    repository: nginx
    tag: alpine
    pullPolicy: IfNotPresent

  config:
    # This generates the default nginx config as per the nextcloud documentation
    default: true
    # custom: |-
    #     worker_processes  1;..

  resources: {}

internalDatabase:
  enabled: false
  name: nextcloud

##
## External database configuration
##
externalDatabase:
  enabled: true

  ## Supported database engines: mysql or postgresql
  type: postgresql

  ## Database host
  host: postgresql.db.svc.cluster.local

  ## Database user
  user: nextcloud

  ## Database password
  password: __pgsql-password__

  ## Database name
  database: nextcloud

  ## Use a existing secret
  existingSecret:
    enabled: false
    # secretName: nameofsecret
    # usernameKey: username
    # passwordKey: password

##
## MariaDB chart configuration
##
mariadb:
  ## Whether to deploy a mariadb server to satisfy the applications database requirements. To use an external database set this to false and configure the externalDatabase parameters
  enabled: false

  auth:
    database: nextcloud
    username: nextcloud
    password: changeme

  architecture: standalone

  ## Enable persistence using Persistent Volume Claims
  ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
  ##
  primary:
    persistence:
      enabled: false
      # storageClass: ""
      accessMode: ReadWriteOnce
      size: 8Gi

##
## PostgreSQL chart configuration
## for more options see https://github.com/bitnami/charts/tree/master/bitnami/postgresql
##
postgresql:
  enabled: false
  postgresqlUsername: nextcloud
  postgresqlPassword: ""
  postgresqlDatabase: master
  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    size: 128Gi
    # storageClass: ""

##
## Redis chart configuration
## for more options see https://github.com/bitnami/charts/tree/master/bitnami/redis
##

redis:
  enabled: true
  auth:
    enabled: true
    password: __redis-password__

## Cronjob to execute Nextcloud background tasks
## ref: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html#webcron
##
cronjob:
  enabled: true
  # Nexcloud image is used as default but only curl is needed
  image: {}
    # repository: nextcloud
    # tag: 16.0.3-apache
    # pullPolicy: IfNotPresent
    # pullSecrets:
    #   - myRegistrKeySecretName
  # Every 5 minutes
  # Note: Setting this to any any other value than 5 minutes might
  #  cause issues with how nextcloud background jobs are executed
  schedule: "*/5 * * * *"
  annotations: {}
  # Set curl's insecure option if you use e.g. self-signed certificates
  curlInsecure: false
  failedJobsHistoryLimit: 5
  successfulJobsHistoryLimit: 2
  # If not set, nextcloud deployment one will be set
  # resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi

  # If not set, nextcloud deployment one will be set
  # nodeSelector: {}

  # If not set, nextcloud deployment one will be set
  # tolerations: []

  # If not set, nextcloud deployment one will be set
  # affinity: {}

service:
  type: ClusterIP
  port: 8080
  loadBalancerIP: nil
  nodePort: nil

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  # Nextcloud Data (/var/www/html)
  enabled: true
  annotations: {}
  ## nextcloud data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"

  ## A manually managed Persistent Volume and Claim
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # existingClaim:

  accessMode: ReadWriteMany
  size: 256Gi
  existingClaim: "nextcloud-shared"

  ## Use an additional pvc for the data directory rather than a subpath of the default PVC
  ## Useful to store data on a different storageClass (e.g. on slower disks)
  nextcloudData:
    enabled: false
    subPath:
    annotations: {}
    # storageClass: "-"
    # existingClaim:
    accessMode: ReadWriteOnce
    size: 8Gi

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

## Liveness and readiness probe values
## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  initialDelaySeconds: 180
  periodSeconds: 10
  timeoutSeconds: 30
  failureThreshold: 10
  successThreshold: 1
readinessProbe:
  enabled: true
  initialDelaySeconds: 180
  periodSeconds: 10
  timeoutSeconds: 30
  failureThreshold: 10
  successThreshold: 1
startupProbe:
  enabled: false
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 30
  successThreshold: 1

## Enable pod autoscaling using HorizontalPodAutoscaler
## ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
##
hpa:
  enabled: true
  cputhreshold: 60
  minPods: 1
  maxPods: 3

nodeSelector: {}

tolerations: []

affinity: {}

## Prometheus Exporter / Metrics
##
metrics:
  enabled: false

  replicaCount: 1
  # The metrics exporter needs to know how you serve Nextcloud either http or https
  https: false
  timeout: 5s

  image:
    repository: xperimental/nextcloud-exporter
    tag: v0.4.0
    pullPolicy: IfNotPresent

  ## Metrics exporter resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  # resources: {}

  ## Metrics exporter pod Annotation and Labels
  # podAnnotations: {}

  # podLabels: {}

  service:
    type: ClusterIP
    ## Use serviceLoadBalancerIP to request a specific static IP,
    ## otherwise leave blank
    # loadBalancerIP:
    annotations:s
      prometheus.io/scrape: "true"
      prometheus.io/port: "9205"
    labels: {}

rbac:
  enabled: false
  serviceaccount:
    create: true
    name: nextcloud-serviceaccount

[amizzo87]

Start with one replica though, with replicaCount: 1; if I start fresh with replicaCount: 3 I get file/permission errors because each replica is trying to set up the nextcloud installation and it conflicts with the other pods’ replicas.

[timtorChen]

Hi @Janl1, I am using this chart with a redis cluster.

In the beginning, I simply set replica: 2, but got the session break every login. After many trial and test, I find we also need to set the php config using redis as a centrial session handler.

phpConfigs:
  redis-cluster.ini: |-
    session.save_handler = rediscluster
    session.save_path = "seed[]=redis-cluster-endpoint&timeout=2&read_timeout=2&failover=error&persistent=1&auth[user]=redis-user&auth[pass]=redis-password"

configs:
  redis-cluster.config.php: |-
    <?php 
    $CONFIG = [
      'memcache.distributed' => '\\OC\\Memcache\\Redis',
      'memcache.locking' => '\\OC\\Memcache\\Redis',
      'redis.cluster' => [
        'seeds' => [
          'redis-cluster-endpoint',
        ],
        'timeout' => 0.0,
        'read_timeout' => 0.0,
        'failover_mode' => \RedisCluster::FAILOVER_ERROR,
        'user' => 'redis-user',
        'password' => 'redis-password'
      ],
    ];

For the config format, maybe check phpredis document, and the nextcloud example. You may also check my WIP PR https://github.com/timtorChen/k3s-garden/pull/222, as I am recently setting the nextcloud HA.

[Janl1]

@timtorChen thank you very much! This actually solved the issue! I somehow thought the Nextcloud redis config would handle the session storage as well.

[2fst4u]

Hi @Janl1, I am using this chart with a redis cluster.

In the beginning, I simply set replica: 2, but got the session break every login. After many trial and test, I find we also need to set the php config using redis as a centrial session handler.

phpConfigs:
  redis-cluster.ini: |-
    session.save_handler = rediscluster
    session.save_path = "seed[]=redis-cluster-endpoint&timeout=2&read_timeout=2&failover=error&persistent=1&auth[user]=redis-user&auth[pass]=redis-password"

configs:
  redis-cluster.config.php: |-
    <?php 
    $CONFIG = [
      'memcache.distributed' => '\\OC\\Memcache\\Redis',
      'memcache.locking' => '\\OC\\Memcache\\Redis',
      'redis.cluster' => [
        'seeds' => [
          'redis-cluster-endpoint',
        ],
        'timeout' => 0.0,
        'read_timeout' => 0.0,
        'failover_mode' => \RedisCluster::FAILOVER_ERROR,
        'user' => 'redis-user',
        'password' => 'redis-password'
      ],
    ];

For the config format, maybe check phpredis document, and the nextcloud example. You may also check my WIP PR https://github.com/timtorChen/k3s-garden/pull/222, as I am recently setting the nextcloud HA.

I'm finally getting somewhere with the issue I'm also having on this. Sorry to bring up a 2 year old issue but I've just found your solution. How would you input this if you don't use redis cluster but instead use a single redis instance?