[Bug]: Primary storage files gone unexpectedly after an upgrade mishap

andreir1723 commented 2 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

The Nextcloud server based on an AWS EC2 instance was cloned, and the new instance was upgraded to Nextcloud version 29.0.1.1. During the upgrade, the old instance was accidentally accessed by a user, and some files were uploaded to the external S3 Wasabi storage. When the new instance was up and running, the uploaded files disappeared from the external storage.

Steps to reproduce

Have 2 Nextcloud instances connected to the S3 external storage as the main storage ready.
Stop one instance and upload files to the external storage on the other instance.
Stop the instance where files were uploaded and start the other instance.
Check whether the uploaded files appear on the new instance.

Expected behavior

All files uploaded to the external storage should appear on the Nextcloud server regardless of the upload method. The files should never be deleted from the external storage without user consent.

Installation method

Community Manual installation with Archive

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 28 to 29)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[ ] Default user-backend (database)
[X] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "cloud2.cs.du.edu",
            "cloud2"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "29.0.1.1",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "upgrade.disable-web": "false",
        "log_type": "file",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "loglevel": "2",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "mail_smtpmode": "smtp",
        "remember_login_cookie_lifetime": "1800",
        "log_rotate_size": "10485760",
        "trashbin_retention_obligation": "auto, 180",
        "versions_retention_obligation": "auto, 365",
        "simpleSignUpLink.shown": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0.5,
            "dbindex": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "csnextcloudlocal",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-east-2.wasabisys.com",
                "region": "us-east-2",
                "port": 443,
                "use_ssl": true,
                "use_path_style": false
            }
        },
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "maintenance": false,
        "default_phone_region": "US",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "updater.release.channel": "stable",
        "maintenance_window_start": 5
    }
}

List of activated Apps

- admin_audit: 1.19.0
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - support: 1.12.0
  - survey_client: 1.17.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - updatenotification: 1.19.1
  - user_ldap: 1.20.0
  - user_status: 1.9.0
  - viewer: 2.3.0
  - workflowengine: 2.11.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No relevant logs available.

Additional info

No response

solracsf commented 2 months ago

Do they share the same database? Because, if not, file should be on the S3 (if the bucket is the same) but not visible on Nextcloud.

joshtrichards commented 1 month ago

Hi @andreir1723,

"objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "csnextcloudlocal",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-east-2.wasabisys.com",
                "region": "us-east-2",
                "port": 443,
                "use_ssl": true,
                "use_path_style": false
            }
        },

All files uploaded to the external storage should appear on the Nextcloud server regardless of the upload method. The files should never be deleted from the external storage without user consent.

You're using S3 as Primary Storage not External Storage. There is a tremendous difference. Please see https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#differences-from-external-storage

During the upgrade, the old instance was accidentally accessed by a user, and some files were uploaded to the external S3 Wasabi storage. When the new instance was up and running, the uploaded files disappeared from the external storage.

Where is your PostgreSQL database for each Nextcloud deployment in this scenario? Is it co-hosted on the same EC2 instances you're cloning?

You pointed two distinct deployments of Nextcloud Server (different versions or not) at the same Primary Storage object store. This is not a reasonable setup.

Particularly if the database is not also shared.

I'm sorry to hear you may have lost some data, but what you described sounds more like a workflow matter and expected behavior for the scenario described, not a bug on the Nextcloud side.

Hope that helps some (and that I'm not misunderstanding your report).

andreir1723 commented 1 month ago

Hi @joshtrichards,

You can categorize it as an upgrade mashup, but it shouldn't have ended with object storage data loss. According to the document you referred to, https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#differences-from-external-storage, "The metadata is only stored in the database and the object store only holds the file content by unique identifier." That's how it's supposed to work. If the metadata in the database is not present, the file content in the object store should still be there. In my case, the file content in the object store is missing. Anyway, thank you for your time looking into the issue.

joshtrichards commented 1 month ago

If the metadata in the database is not present, the file content in the object store should still be there.

Only in a static situation.

E.g. if you lose your database, your file content will be there in the bucket. That's not what happened.

The Nextcloud you brought up thinks it's authoritative for whatever is in that bucket. Just as authoritative as the other one. There will be overlapping file ids. File actions will be associated with the wrong file content.

As @solracsf suggested, the content written by the other Nextcloud may still be in the bucket itself, but it will not appear in Nextcloud. Or, if it does, it's likely associated with the wrong metadata. If you are lucky enough that the overlapping bits haven't lead to destructive actions against file content, it may be there, but that's certainly not guaranteed.

How are you determining that the file content is missing anyhow?

andreir1723 commented 1 month ago

OK. For example, I can query the database on the backed-up instance and see a file "userID/files/myfile.txt" with urn:id:77777.

If I query the database on the upgraded instance, I don't see a file with the filename "myfile.txt" whatsoever. If that file were assigned a different ID or if urn:id:77777 were assigned to a different file, I'd agree with your overlapping file IDs hypothesis. However, urn:id:77777 is not present either.

To determine if the content is present on S3, I do a direct query to the object storage. No trace of the file is found there.

andreir1723 commented 1 month ago

Do they share the same database? Because, if not, file should be on the S3 (if the bucket is the same) but not visible on Nextcloud.

That is the expected behavior, but it is not what happened in my case. Thank you for your feedback, @solracsf.

kesselb commented 1 month ago

urn:id:77777

77777 is the fileid from the oc_filecache table.

If you find higher fileids in the table, then it was most likely an overlapping id.

andreir1723 commented 1 month ago

@kesselb Overlapping IDs should not remove actual data from the S3 storage. An overlapping ID would make the file invisible on the Nextcloud server, and that would be acceptable. However, the problem is that the file does not exist on the storage anymore.

kesselb commented 1 month ago

Do you find, on the new node, higher file ids?

It's possible that files are generated during the upgrade and stored in appdata. That's also on objectstore and the file might be deleted right after the upgrade.

nextcloud-command commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

nextcloud / server