search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.36k stars 4.06k forks source link

Expiring versions does not work for deleted files #40450

Open arnowelzel opened 1 year ago

arnowelzel commented 1 year ago

⚠️ This issue respects the following points: ⚠️

Bug description

When Nextcloud tries to expire versions for files which got deleted, it will fail because it will try to find the original file which does not exist any longer. This is what I get when I try to run that for a specific user (all private data anonymized):

occ versions:expire USERNAME -vvv
Remove deleted files of   USERNAME

In Root.php line 206:

  [OCP\Files\NotFoundException]
  /path-example/filename

Exception trace:
  at /var/www/server.example/lib/private/Files/Node/Root.php:206
 OC\Files\Node\Root->get() at /var/www/server.example/lib/private/Files/Node/Folder.php:135
 OC\Files\Node\Folder->get() at /var/www/server.example/apps/files_versions/lib/Storage.php:595
 OCA\Files_Versions\Storage::OCA\Files_Versions\{closure}() at n/a:n/a
 array_filter() at /var/www/server.example/apps/files_versions/lib/Storage.php:610
 OCA\Files_Versions\Storage::expireOlderThanMaxForUser() at /var/www/server.example/apps/files_versions/lib/Command/ExpireVersions.php:110
 OCA\Files_Versions\Command\ExpireVersions->expireVersionsForUser() at /var/www/server.example/apps/files_versions/lib/Command/ExpireVersions.php:86
 OCA\Files_Versions\Command\ExpireVersions->execute() at /var/www/server.example/3rdparty/symfony/console/Command/Command.php:298
 Symfony\Component\Console\Command\Command->run() at /var/www/server.example/3rdparty/symfony/console/Application.php:1040
 Symfony\Component\Console\Application->doRunCommand() at /var/www/server.example/3rdparty/symfony/console/Application.php:301
 Symfony\Component\Console\Application->doRun() at /var/www/server.example/3rdparty/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /var/www/server.example/lib/private/Console/Application.php:211
 OC\Console\Application->run() at /var/www/server.example/console.php:100
 require_once() at /var/www/server.example/occ:11

versions:expire [<user_id>...]

Steps to reproduce

  1. create file
  2. make changes to, so that there are multiple versions
  3. delete the file
  4. run occ versions:expire

Note: I don't know if this will only happen, if there is at least one old version to be deleted.

Expected behavior

occ versions:expire and the respective background job work without any error message.

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

27.0.2, 27.1.0, 27.1.1, 27.1.3

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 22.1 to 22.2)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.0x0c.de"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "",
        "overwrite.cli.url": "https:\/\/nextcloud.0x0c.de",
        "dbtype": "mysql",
        "version": "27.0.2.1",
        "installed": true,
        "htaccess.RewriteBase": "\/",
        "maintenance": false,
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0
        },
        "knowledgebaseenabled": false,
        "theme": "",
        "simpleSignUpLink.shown": false,
        "loglevel": 2,
        "log_authfailip": true,
        "mysql.utf8mb4": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 180",
        "app_install_overwrite": [
            "apporder",
            "deck",
            "keeweb",
            "appointments",
            "drawio",
            "bookmarks",
            "spreed",
            "unroundedcorners",
            "fulltextsearch",
            "fulltextsearch_elasticsearch",
            "files_fulltextsearch",
            "files_fulltextsearch_tesseract",
            "passwords",
            "gpxpod",
            "dicomviewer",
            "files_bpm"
        ],
        "has_rebuilt_cache": true,
        "default_language": "de",
        "default_phone_region": "DE",
        "check_data_directory_permissions": false,
        "log_rotate_size": 52428800,
        "profile.enabled": false,
        "preview_max_x": 1920,
        "preview_max_y": 1920,
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_smtpport": "465",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "reference_opengraph": false
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - bruteforcesettings: 2.7.0
  - calendar: 4.5.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - collectives: 2.7.1
  - comments: 1.17.0
  - contacts: 5.4.0
  - contactsinteraction: 1.8.0
  - dav: 1.27.0
  - deck: 1.11.0
  - drawio: 2.1.2
  - external: 5.2.1
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_bpm: 1.2.0
  - files_pdfviewer: 2.8.0
  - files_photospheres: 1.27.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - keeweb: 0.6.13
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - news: 23.0.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - notify_push: 0.6.3
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - previewgenerator: 5.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - richdocuments: 8.1.1
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.14.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_webauthn: 1.2.0
  - updatenotification: 1.17.0
  - viewer: 2.1.0
  - workflowengine: 2.9.0
Disabled:
  - dashboard: 7.7.0 (installed 7.7.0)
  - encryption: 2.15.0 (installed 2.13.0)
  - files_external: 1.19.0 (installed 1.19.0)
  - recommendations: 1.6.0 (installed 0.4.0)
  - related_resources: 1.2.0 (installed 1.1.0-alpha1)
  - suspicious_login: 5.0.0
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0
  - user_status: 1.7.0 (installed 1.5.0)
  - weather_status: 1.7.0 (installed 1.3.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

arnowelzel commented 1 year ago

Just for the records - the problem also exists in Nextcloud 27.1.0.

Forza-tng commented 1 year ago

Just for the records - the problem also exists in Nextcloud 27.1.0.

Notice the same issue. Does it mean it won't expire any files when it occurs?

arnowelzel commented 1 year ago

Just for the records - the problem also exists in Nextcloud 27.1.0.

Notice the same issue. Does it mean it won't expire any files when it occurs?

Correct - as soon as there is only one deleted file with versions left, the whole expire process won't work any longer.

arnowelzel commented 1 year ago

just for the records - the problem also exists in Nextcloud 27.1.1.

kesselb commented 1 year ago

just for the records - the problem also exists in Nextcloud 27.1.1.

The next time, update the initial post instead of writing a comment.

arnowelzel commented 1 year ago

just for the records - the problem also exists in Nextcloud 27.1.1.

The next time, update the initial post instead of writing a comment.

Done

geez0x1 commented 1 year ago

Just a little addition. I was seeing this on a few files, and as a test I decided to re-create them, wait for the cronjob to run (may not be necessary), and then remove them again. Stopped seeing it. Also ran versions:expire --verbose to double check, no such errors reported there or in the log. (NC 27.1.3)

Secondly I tried creating a new test file, modifying it a few times, checking versions were indeed present, and then deleting it. Running versions:expire --verbose again, no errors were produced. So it seems possible that these issues were caused at some point in a previous version, but only encountered now (i.e., unexpected data). The underlying reason for them might have disappeared already. Nevertheless, the code could be made more robust against this.

arnowelzel commented 1 year ago

Just a little addition. I was seeing this on a few files, and as a test I decided to re-create them, wait for the cronjob to run (may not be necessary), and then remove them again. Stopped seeing it. Also ran versions:expire --verbose to double check, no such errors reported there or in the log. (NC 27.1.3)

Secondly I tried creating a new test file, modifying it a few times, checking versions were indeed present, and then deleting it. Running versions:expire --verbose again, no errors were produced. So it seems possible that these issues were caused at some point in a previous version, but only encountered now. The underlying reason for them might have disappeared already. Nevertheless, the code could be made more robust against this.

Still it happens here with NC 27.1.3:

   32 [------------------>---------] < 1 sec
In Root.php line 206:

  [OCP\Files\NotFoundException]
  /SomeUser/files/somefolder/somefile.example

Exception trace:
  at /var/nextcloud-root-folder/lib/private/Files/Node/Root.php:206
 OC\Files\Node\Root->get() at /var/nextcloud-root-folder/lib/private/Files/Node/Folder.php:135
 OC\Files\Node\Folder->get() at /var/nextcloud-root-folder/apps/files_versions/lib/Storage.php:595
 OCA\Files_Versions\Storage::OCA\Files_Versions\{closure}() at n/a:n/a
 array_filter() at /var/nextcloud-root-folder/apps/files_versions/lib/Storage.php:610
 OCA\Files_Versions\Storage::expireOlderThanMaxForUser() at /var/nextcloud-root-folder/apps/files_versions/lib/Command/ExpireVersions.php:110
 OCA\Files_Versions\Command\ExpireVersions->expireVersionsForUser() at /var/nextcloud-root-folder/apps/files_versions/lib/Command/ExpireVersions.php:97
 OCA\Files_Versions\Command\ExpireVersions->OCA\Files_Versions\Command\{closure}() at /var/nextcloud-root-folder/lib/private/User/Manager.php:632
 OC\User\Manager->callForSeenUsers() at /var/nextcloud-root-folder/apps/files_versions/lib/Command/ExpireVersions.php:98
 OCA\Files_Versions\Command\ExpireVersions->execute() at /var/nextcloud-root-folder/3rdparty/symfony/console/Command/Command.php:298
 Symfony\Component\Console\Command\Command->run() at /var/nextcloud-root-folder/3rdparty/symfony/console/Application.php:1040
 Symfony\Component\Console\Application->doRunCommand() at /var/nextcloud-root-folder/3rdparty/symfony/console/Application.php:301
 Symfony\Component\Console\Application->doRun() at /var/nextcloud-root-folder/3rdparty/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /var/nextcloud-root-folder/lib/private/Console/Application.php:211
 OC\Console\Application->run() at /var/nextcloud-root-folder/console.php:100
 require_once() at /var/nextcloud-root-folder/occ:11
arnowelzel commented 1 year ago

In addition - it seems the problem is triggered by this:

  1. A file gets created in data/user/files.
  2. The file gets changed, so there is at least one additional version in data/user/files_version.
  3. The file gets deleted, so the original file in data/user/files does not exist any linger.

Now the version expiration will still see the old version(s) in data/user/files_version and expects the original file to be present in data/user/files.

The expected behaviour should be, that old version(s) in data/user/files_version should just get deleted if the original file does not exist any longer.

Forza-tng commented 1 year ago

The expected behaviour should be, that old version(s) in data/user/files_version should just get deleted if the original file does not exist any longer.

Only if retention period has been reached, like for other files, so that the file could be recovered according to those expectations.

arnowelzel commented 1 year ago

Only if retention period has been reached, like for other files, so that the file could be recovered according to those expectations.

How to recover a deleted file in Nextcloud if it is not in the trash folder but only in file versions?