Versioning ignores versions_retention_obligation `D, auto`

[GAS85]

Steps to reproduce

1. Configure versions_retention_obligation to .e.g. "versions_retention_obligation": "14, auto", as described here https://github.com/nextcloud/server/blob/master/config/config.sample.php#L603.

   * * ``D, auto``
   *     keep versions at least for D days, apply expire rules to all versions
   *     that are older than D days

2. Create file, e.g. todo.txt
3. Update this file few times
4. See that older version are deleted, but they must be kept for a configured (here 14) amount of days.

Expected behaviour

Older files versions must be handled as it is configured in config.php and as it is described in a Doku.

Actual behaviour

Older files versions has being deleted independent of settings in config.php:

Server configuration

Operating system: Ubuntu 18.04

Web server: Apache 2.4.41

Database: 10.1.44-MariaDB

PHP version: 7.3.15

Nextcloud version: 17.0.3

Updated from an older Nextcloud/ownCloud or fresh install: update

Where did you install Nextcloud from: Official

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Enabled: - accessibility: 1.3.0 - activity: 2.10.1 - admin_audit: 1.7.0 - audioplayer: 2.9.0 - bruteforcesettings: 1.5.0 - calendar: 2.0.2 - checksum: 0.4.4 - cloud_federation_api: 1.0.0 - comments: 1.7.0 - data_request: 1.4.0 - dav: 1.13.0 - deck: 0.8.0 - federatedfilesharing: 1.7.0 - federation: 1.7.0 - files: 1.12.0 - files_automatedtagging: 1.7.0 - files_external: 1.8.0 - files_pdfviewer: 1.6.0 - files_retention: 1.6.0 - files_rightclick: 0.15.1 - files_sharing: 1.9.0 - files_trashbin: 1.7.0 - files_versions: 1.10.0 - files_videoplayer: 1.6.0 - firstrunwizard: 2.6.0 - flowupload: 0.1.8 - gallery: 18.4.0 - gpxpod: 4.1.1 - keeweb: 0.6.2 - logreader: 2.2.0 - lookup_server_connector: 1.5.0 - mail: 1.1.3 - maps: 0.1.5 - nextcloud_announcements: 1.6.0 - notes: 3.1.5 - notifications: 2.5.0 - oauth2: 1.5.0 - ocdownloader: 1.7.6 - password_policy: 1.7.0 - phonetrack: 0.6.2 - polls: 1.3.0 - previewgenerator: 2.2.0 - privacy: 1.1.0 - provisioning_api: 1.7.0 - radio: 0.6.6 - recommendations: 0.5.0 - serverinfo: 1.7.0 - sharebymail: 1.7.0 - survey_client: 1.5.0 - systemtags: 1.7.0 - text: 1.1.1 - theming: 1.8.0 - twofactor_backupcodes: 1.6.0 - twofactor_totp: 4.1.2 - unsplash: 1.1.5 - updatenotification: 1.7.0 - viewer: 1.2.0 - weather: 1.7.1 - workflowengine: 1.7.0 Disabled: - encryption - impersonate - sharerenamer - support - user_ldap ```

Nextcloud configuration:

Config report

``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": { "0": "***REMOVED SENSITIVE VALUE***", "2": "***REMOVED SENSITIVE VALUE***" }, "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/nextcloud", "dbtype": "mysql", "version": "17.0.3.1", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "filesystem_check_changes": 0, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.distributed": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379, "timeout": 1.5 }, "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "theme": "", "logfile": "\/var\/nextcloud\/data\/nextcloud.log", "loglevel": 1, "trashbin_retention_obligation": "14, auto", "versions_retention_obligation": "14, auto", "data-fingerprint": "***REMOVED SENSITIVE VALUE***", "enable_previews": true, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\Movie", "OC\\Preview\\PDF", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown" ], "preview_max_x": 1080, "preview_max_y": 1920, "auth.bruteforce.protection.enabled": true, "simpleSignUpLink.shown": false, "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "twofactor_enforced": "true", "twofactor_enforced_groups": [ "admin" ], "twofactor_enforced_excluded_groups": [], "has_rebuilt_cache": true, "updater.release.channel": "stable", "app_install_overwrite": [ "keeweb", "radio" ] } } ```

Are you using external storage, if yes which one: local smb

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"} {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"} {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"} {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401886","userAgent":"--","version":"17.0.3.1"} {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:40+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401876","userAgent":"--","version":"17.0.3.1"} {"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:40+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401859","userAgent":"--","version":"17.0.3.1"} ```

[Peac]

Same here on 18.0.3.

I configured my Nextcloud to keep all versions for a month and apply the auto rule afterwards. When the day came, that I needed a specific version, it was gone. Quite a problem, if you trust that a documented feature works as it should.

[szaimen]

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

[GAS85]

For NC 21 please check it, I did not perform update. For NC 20.0.10 still the Problem. E.g. 10 Days ago was 3 updates performed:

But only 1 Version saved:

According to config, all versions for last 14 days should be saved:

[szaimen]

Hey @icewind1991, for me the reported bug looks like documentation is missing about the versions_retention_obligation setting because it seems to work correctly but missing some details: For ‘D, auto‘ it seems to clear some versions in between (probably due to background jobs) and all remaining versions after the D setting expires. If this should be true, I would create a PR that makes this clear. Would be great if you could reply if this is working as intended or a bug. Thanks! :)

[GAS85]

it seems to clear some versions in between (probably due to background jobs) and all remaining versions after the D setting expires

Actually not, you can see on screenshot that for the rule 14, auto I have versions that are not saved during last 14 days but versions beyond working fine. On a new screenshot you can see (I hope, have to unzoom it to make sceenshot) that even versions around 1 year old are presented, but "keep it all for at least 14 days" not respected.

Screenshot


What you are pointing is a valid use case, but should be configured via auto, D instead, as per https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#file-versions

[szaimen]

Thanks for the clarification! So yes, this smells like a bug.

[szaimen]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

[Peac]

I'm on 25.0.2. It's still not working.

What I did:

1. Settings: '30, auto'
2. Created 10 versions of a file
3. Verified that all versions were shown
4. Waited five minutes (till cron started)
5. Two versions remained

[Andeye]

We've encountered the same issue. I've verified that the bug exists in both NC versions 23.0.12 and 26.0.0.11.

The problem seems to be somewhere in files_versions/lib/Storage.php or perhaps apps/files_versions/lib/Expiration.php.

Most likely getAutoExpireList() should not be called here, https://github.com/nextcloud/server/blob/v23.0.8/apps/files_versions/lib/Storage.php#L643 , or then getAutoExpireList() should take isExpired() in consideration. It's also possible that the expire() should return instead of entering the foreach loop: https://github.com/nextcloud/server/blob/v23.0.8/apps/files_versions/lib/Storage.php#L835

The tests have expiration data for testing the rules, but doesn't seem to actually test the functionality properly: https://github.com/nextcloud/server/blob/master/apps/files_versions/tests/ExpirationTest.php

(My apologies that most links point to an old version. But nothing related to this issue seems to have changed in the newer versions, and the issue still persists.)