[Bug]: Files are silently removed from database when storage fails

nfp0 commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

When trying to access a file on Nextcloud but, for some reason, the underlying storage device fails with an I/O error, Nextcloud removes the file from the database silently.

For an unsuspecting user, after solving the storage device problem and restarting Nextcloud, it will find said file has disappeared even though it still exists on the Nextcloud data directory. This happens silently, so a user might find out months later that many of his files have mysteriously vanished.

The files can be added back by re-scanning the data directory, but that is assuming the user ever finds out that these files have been disappearing. There are no Warning or Error logs about this either.

I could only trigger this bug by downloading big folders with multiple files through the web interface and having a storage I/O error during the download, but there might be other ways to trigger it.

The behavior seems similar to the one described here: https://help.nextcloud.com/t/content-wiped-out-from-database-after-storage-unmounted/89534

Steps to reproduce

I'll describe how I can trigger the bug consistently on an external HDD, but there might be other ways to trigger it.

Have Nextcloud with the data directory on an external USB HDD.
Start the download of a big folder with multiple files through the web interface.
Pull the USB cable to simulate a storage failure.
Re-plug the cable, re-mount the storage, restart Nextcloud and refresh the web page.
Notice how at least one file has vanished from the folder you were trying to download (possibly the file that was being accessed at the instant you unplugged the HDD).

Expected behavior

Nexcloud should provide an error message in the log, but should never assume a file has been deleted just because of an I/O error.

Installation method

Community Docker image

Operating system

Other

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[x] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.example.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.3.2",
        "overwrite.cli.url": "https:\/\/nextcloud.example.com",
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "default_phone_region": "PT",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 2,
        "maintenance": false,
        "theme": ""
    }
}

List of activated Apps

Enabled:
  - activity: 2.17.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - nextcloud_announcements: 1.14.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - recommendations: 1.4.0
  - related_resources: 1.0.4
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - support: 1.8.0
  - survey_client: 1.13.0
  - systemtags: 1.15.0
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - twofactor_totp: 7.0.0
  - updatenotification: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
Disabled:
  - admin_audit
  - bruteforcesettings
  - encryption
  - files_external
  - suspicious_login
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No logs are printed when triggering this bug.

Additional info

OS: Arch Linux

nfp0 commented 1 year ago

Edited the post to add the selection details, as it seems GitHub deselected them all when it reloaded the page because of a character limit.

nfp0 commented 1 year ago

@szaimen I have noticed you added the 25-feedback tag. Does this mean that additional feedback is needed from me? Is anything missing in the report?

szaimen commented 1 year ago

Does this mean that additional feedback is needed from me?

No. All good!

jakobkrieger commented 1 year ago

Is there any progress on this issue?

I just had this issue when an external storage folder, which is mounted via Samba CIFS failed because of downtime on the storage box provider's end and when triggering a occ files:scan the external storage folder appeared again, but all files were reindexed so the Desktop Client removed all of them and redownloads them again, which can hardly be the expected behavior

I'd be very glad if this could be addressed, like checking if an external storage is mounted/reachable and not just deleting the files therein from the database, which is my current understanding of the mechanism :)

nfp0 commented 1 year ago

@jakobkrieger I did not know it triggered the Desktop Client to redownload all the files again, as I was not using using that client. But that makes this issue even worse than I thought! :frowning_face:

nextcloud / server