[Bug]: Automatic (not on-demand) preview generation fills disk

csware commented 1 month ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

After upgrading to 29 I noticed that my previews folder grew over 250 GiB. I tried to investigate and found that Nextcloud automatically generates previews after an upload (using Nextcloud client) and not just on-demand (using the browser interface) as the old versions did. (I also have the impression that Nextcloud generated previews for all files with missing previews after the upgrade to 29.)

Maybe this is related to the new FilesMetadata API?

Steps to reproduce

Upload a file using Nextcloud client

Expected behavior

No previews are generated automatically. I've many users who do not use the browser interface and, thus, generating lots pf previews is a waste of CPU time and space.

Generating previews only happens on-demand or can be configured to be on-demand only.

Installation method

Community Manual installation with Archive

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.1.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "loglevel": 2,
        "integrity.check.disabled": false,
        "apps_paths": [
            {
                "path": "\/var\/www\/XXXXXXX\/apps",
                "url": "\/apps",
                "writable": true
            }
        ],
        "maintenance": false,
        "theme": "",
        "overwrite.cli.url": "https:\/\/XXXXXXX",
        "overwriteprotocol": "https",
        "activity_expire_days": 31,
        "trusted_domains": [
            "XXXXXXX"
        ],
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto",
        "htaccess.RewriteBase": "\/",
        "mail_smtpmode": "sendmail",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "appstore.experimental.enabled": false,
        "singleuser": false,
        "log_rotate_size": 104857600,
        "logtimezone": "Europe\/Berlin",
        "mail_sendmailmode": "pipe",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        },
        "mysql.utf8mb4": true,
        "defaultapp": "files",
        "updater.release.channel": "stable",
        "default_phone_region": "DE",
        "mail_smtpauthtype": "PLAIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "maintenance_window_start": 3,
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "enabledPreviewProviders": [
            "OC\\Preview\\Imaginary"
        ],
        "preview_max_x": 1024,
        "preview_max_y": 1024
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.4
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.0
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - gpxpod: 5.0.18
  - impersonate: 1.16.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - onlyoffice: 9.2.2
  - polls: 7.0.3
  - provisioning_api: 1.19.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - spreed: 19.0.1
  - systemtags: 1.19.0
  - tasks: 0.16.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - encryption: 2.17.0
  - password_policy: 1.19.0 (installed 2.0.0)
  - photos: 2.5.0 (installed 2.5.0)
  - privacy: 1.13.0 (installed 1.0.0)
  - recommendations: 2.1.0 (installed 0.4.0)
  - support: 1.12.0 (installed 1.0.0)
  - survey_client: 1.17.0 (installed 1.4.0)
  - suspicious_login: 7.0.0
  - twofactor_totp: 11.0.0-dev
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

kesselb commented 1 month ago

Can confirm.

Steps to reproduce:

Connect desktop client to Nextcloud
Copy an image
See the post request for the bulk upload endpoint
Max Preview + 256er Previews are generated

To monitor the preview folder for create events: inotifywait -m -r -e create /path/to/preview/folder

Output:

preview/1/0/ CREATE,ISDIR a
preview/1/0/a/b/2/6/9/39539/ CREATE 2560-1440-max.jpg
preview/1/0/a/b/2/6/9/39539/ CREATE 256-144.jpg
preview/b/5/7/ CREATE,ISDIR 7
preview/b/5/7/7/2/e/7/39548/ CREATE 1920-1080-max.jpg
preview/b/5/7/7/2/e/7/39548/ CREATE 256-144.jpg
preview/e/4/ CREATE,ISDIR 9
preview/e/4/9/f/1/7/b/39556/ CREATE 2559-1599-max.jpg
preview/e/4/9/f/1/7/b/39556/ CREATE 256-160.jpg
preview/c/a/b/ CREATE,ISDIR 8
preview/c/a/b/8/c/3/7/39565/ CREATE 1920-1080-max.jpg
preview/c/a/b/8/c/3/7/39565/ CREATE 256-144.jpg
preview/7/1/ CREATE,ISDIR c
preview/7/1/c/7/2/2/e/39573/ CREATE 1920-1200-max.png
preview/7/1/c/7/2/2/e/39573/ CREATE 256-160.png

That could be https://github.com/nextcloud/server/blob/4e2ada015528ca37736f142f2456c8375ab0d125/lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php#L73, however adding a return before getPreview had no effect on my end.

kesselb commented 1 month ago

cc @ArtificialOwl @artonge

marcos-guerrero commented 2 weeks ago

What is worse is that even with enable_previews => false in config.php the preview folder is growing too I dropped 4 images on my desktop app and

root@maqueta-nextcloud-backend2:~# inotifywait -m -r -e create /disco_datos_nextcloud/nextcloud_data/appdata_XXX/preview/
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/disco_datos_nextcloud/nextcloud_data/appdata_XXX/preview/5/6/1/ CREATE,ISDIR f
/disco_datos_nextcloud/nextcloud_data/appdata_XXX/preview/0/3/9/ CREATE,ISDIR 7
/disco_datos_nextcloud/nextcloud_data/appdata_XXX/preview/4/7/a/f/ CREATE,ISDIR d
/disco_datos_nextcloud/nextcloud_data/appdata_XXX/preview/7/f/8/2/ CREATE,ISDIR f

I'm on nc 29.0.2

kesselb commented 2 weeks ago

What is worse is that even with enable_previews => false in config.php the preview folder is growing too

https://github.com/nextcloud/server/issues/45697

artonge commented 2 weeks ago

We indeed need to generate the blurhash after the preview generation:

emit an event when a preview is created with the original and preview File objects https://github.com/nextcloud/server/blob/0ddfe9ed6e1d857e2058a71d75940ebba3748461/lib/private/Preview/Generator.php#L77-L87
in the listener generate and update the blurhash metadata

csware commented 2 weeks ago

@artonge I may have miss understood your last comment. There is a difference between "we need to generate the blurhash after preview generation" and "we need to always generate a blurhash (after causing a preview generation)"

artonge commented 1 week ago

I may have miss understood your last comment. There is a difference between "we need to generate the blurhash after preview generation" and "we need to always generate a blurhash (after causing a preview generation)"

To be clearer, blurhash generation should not trigger preview generation ;)

nextcloud / server