Kubernetes - Redeploying not working when using S3 as default storage backend

[LunaticZorr]

Nextcloud version (eg, 12.0.2): 16.0.4 Operating system and version (eg, Ubuntu 17.04): Kubernetes / Docker Apache or nginx version (eg, Apache 2.4.25): Docker Image nextcloud:16.0.4-apache PHP version (eg, 7.1): Docker Image nextcloud:16.0.4-apache

The issue you are facing:

• We are deploying Nextcloud on Kubernetes using the Helm Chart from https://github.com/helm/charts/tree/master/stable/nextcloud .
• We changed the Docker image to be nextcloud:16.0.4-apache
• We use the s3.config.php option to store our files on S3.
• We use the external database option to use a MariaDB server we already have.

We launch Nextcloud the first time and it creates the DB correctly, creates the first user correctly, and starts up as expected. We can login, and create / upload files. To verify our files are secure and retrievable after a major failure, we re-deploy the Nextcloud deployment (scale to 0, scale to 1). After this, while starting the logs show the following:

Initializing nextcloud 16.0.4.1 ...
Initializing finished
New nextcloud instance
Installing with MySQL database
starting nextcloud installation
The username is already being used
retrying install...
The username is already being used
retrying install...
The username is already being used
retrying install...

This is the first issue. WHY does it try to re-install? The Database is still there, and so is the previous user. Why does it not just connect and re-use what is there?

After a couple of minutes the container dies and starts again, this time without failure. BUT, when trying to browse to nextcloud, we are greeted with the following message:

Error
It looks like you are trying to reinstall your Nextcloud. However the file CAN_INSTALL is missing from your config directory. Please create the file CAN_INSTALL in your config folder to continue.

If I create the CAN_INSTALL file, I am prompted with the installation/setup screen and am told that the admin account I want to use does already exist.

Is this the first time you've seen this error? (Y/N): Y

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'debug' => true,
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'objectstore' => 
  array (
    'class' => '\\OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => 'nextcloud-files',
      'autocreate' => true,
      'key' => '**************',
      'secret' => '****************',
      'region' => 'eu-west-1',
      'use_ssl' => true,
    ),
  ),
  'passwordsalt' => '********************',
  'secret' => '******************',
  'trusted_domains' => 
  array (
    0 => 'localhost',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '16.0.4.1',
  'overwrite.cli.url' => 'http://localhost',
  'dbname' => 'nextcloud',
  'dbhost' => 'mariadb.mariadb',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'oc_user129',
  'dbpassword' => '*****************',
  'instanceid' => '************',
);

Any idea on how to solve this issue?

[cuihaikuo]

Same problem!

[fle108]

exactly the same problem even with manual yaml manifest (not helm) and persistentvolumeclaims. I tested to delete the user in database, I works for this step but the next problem is :

Command "maintenance:install" is not defined.

  Did you mean one of these?
      app:install
      maintenance:data-fingerprint
      maintenance:mimetype:update-db
      maintenance:mimetype:update-js
      maintenance:mode
      maintenance:repair
      maintenance:theme:update
      maintenance:update:htaccess

retrying install...

[JasperZ]

Just out of curiosity what do you use as storage backend? I never got it working with nfs backed persistent volumes. The rsync happening in the entrypoint.sh didn't fully finish for some reason. Also it took a pretty long time until the "install" was finished. And when I killed the pod the new one was trying to install nextcloud again.

[fle108]

Just out of curiosity what do you use as storage backend? I never got it working with nfs backed persistent volumes. The rsync happening in the entrypoint.sh didn't fully finish for some reason. Also it took a pretty long time until the "install" was finished. And when I killed the pod the new one was trying to install nextcloud again.

hi, I use azure file storage but I've to mount it with specific mount options (uid 33 for www-data) in my persistentVolume manifest otherwise it doesn't work

  mountOptions:
  - dir_mode=0770
  - file_mode=0770
  - uid=33
  - gid=33

actually I battle with initContainers to be able to push ConfigMap file (.user.ini) to set up php options like upload_max_filesize

sources: https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/37 https://docs.nextcloud.com/server/13.0.0/admin_manual/configuration_files/big_file_upload_configuration.html

[GoingOffRoading]

Anybody figure out a fix to this problem?

I have all data persisted on a NAS and just wiped my Kubernetes host to re-start my containers from scratch.

When I launch NExtCloud, I get the same "It looks like you are trying to reinstall your Nextcloud. However the file CAN_INSTALL is missing from your config directory. Please create the file CAN_INSTALL in your config folder to continue." error.

I can not find any documentation on this nor many other threads.

[kquinsland]

@GoingOffRoading You need to make sure that instanceid persists across the pod lifecycle. Do this by making sure that /var/www/html/config is on a persistent vol.

see: https://github.com/nextcloud/docker/issues/1006#issuecomment-682276451

[robertoschwald]

This seems to be a bug when you use S3 as primary storage. We don't want to have any persistent storages at all but use S3, only.

[agowa]

@robertoschwald just get the file out of the config folder and put it into your helm chart as well?

Or use an init container that creates that file from something you stored in S3...

[jessebot]

For right now, you still need a persistent volume for the config directory as well, even when using S3. That's been my experience, at least. You can set persistence in the helm chart, but we probably still need to separate out the config dir persistence entirely from the data dir. I'll see if I can find the other issue mentioning this and link it back here.

[agowa]

you still need a persistent volume for the config directory as well

you don't configmap works as well at least last time I checked the files within the config folder weren't dynamically updated at runtime by the application itself...

And alternatively an init container could just bootstrap the config directory using a script or something...

[jessebot]

I haven't tested this in about 6 months, but I thought there was something that changed in the config directory that prevented this from working. I can't remember what it was though. Oh, maybe it was the /var/www/html directory itself or something else in the /var/www/html/data directory?

Either way, I haven't had time to test this again in a while, so I'm open to anyone else in the community testing installing the latest version of this helm chart, enabling s3 as the default storage backend via a the nextcloud.configs parameter (which should create a ConfigMap), and verifying it's still broken. If it is broken still, we need to know precisely which directory needs to be persisted to fix this and why. From there, we can figure out what needs to be done, including the suggestions you've made, @agowa to see if there's anything we can mount or script away to solve this. 🙏