[Bug]: Cronjob went OOM once a day, I suspect GenerateMetadataJob.

Waldmeisda commented 6 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Hi, I've discovered that my nextcloud cronjob that runs every 15 minutes crashes ones per day 16:45 I'm running the cron with 1G memory limit to evoid the oom killer kick in: php8.1 -d memory_limit=1G -f /var/www/nextcloud/cron.php And get an email everyday: PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20814848728 bytes) in /var/www/nextcloud/lib/private/Files/Storage/Local.php on line 327

grafik I'm running the current 28.0.1

Any suggestion howto futher debug what job exactly caused the issue?

Since it happens once every day at a specific timestamp it should be one of those: grafik

Steps to reproduce

Create Crontab with the current release: /15 * php8.1 -d memory_limit=1G -f /var/www/nextcloud/cron.php

Once per day, cron.php grows to 20GB Memory --> OOM

Expected behavior

Cronjob should not run OOM

Installation method

None

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
           ***REMOVED SENSITIVE VALUE***   
        ],
        "default_phone_region": "DE",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/cloud.xxxx.com",
        "dbtype": "mysql",
        "version": "28.0.1.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/var\/log\/nextcloud.log",
        "loglevel": 0,
        "log_rotate_size": "104857600",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "theme": "",
        "maintenance": false,
        "secret": "***REMOVED SENSITIVE VALUE***",
        "updatechecker": false,
        "trashbin_retention_obligation": "auto",
        "appstoreurl": "https:\/\/apps.nextcloud.com\/api\/v1",
        "appstoreenabled": true,
        "mysql.utf8mb4": true,
        "filelocking.enabled": "true",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "updater.release.channel": "stable",
        "app_install_overwrite": [
            "bookmarks"
        ],
        "memories.exiftool": "\/var\/www\/owncloud\/apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/owncloud\/apps\/memories\/bin-ext\/go-vod-amd64"
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - audioplayer: 3.4.1
  - bookmarks: 13.1.3
  - bruteforcesettings: 2.8.0
  - calendar: 4.6.3
  - circles: 28.0.0-dev
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contacts: 5.5.1
  - contactsinteraction: 1.9.0
  - dashboard: 7.8.0
  - dav: 1.29.1
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_automatedtagging: 1.18.0
  - files_external: 1.20.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - keeweb: 0.6.15
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - mail: 3.5.2
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.2
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - richdocuments: 8.3.0
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - support: 1.11.0
  - survey_client: 1.16.0
  - systemtags: 1.18.0
  - tasks: 0.15.0
  - text: 3.9.1
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - updatenotification: 1.18.0
  - viewer: 2.2.0
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - admin_audit: 1.18.0
  - checksum: 1.2.2 (installed 1.2.2)
  - duplicatefinder: 1.1.4 (installed 1.1.4)
  - encryption: 2.16.0
  - files_accesscontrol: 1.18.0 (installed 1.18.0)
  - files_retention: 1.16.0 (installed 1.16.0)
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - firstrunwizard: 2.17.0 (installed 2.17.0)
  - groupfolders: 16.0.1 (installed 16.0.1)
  - memories: 6.2.2 (installed 6.2.2)
  - polls: 6.0.1 (installed 6.0.1)
  - previewgenerator: 5.4.0 (installed 5.4.0)
  - spreed: 18.0.1 (installed 18.0.1)
  - suspicious_login: 6.0.0
  - twofactor_totp: 10.0.0-beta.2
  - user_ldap: 1.19.0
  - user_status: 1.8.1 (installed 1.8.1)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"EgNErFpZbZCmFkP57K24","level":0,"time":"2024-01-14T15:45:09+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Run OC\\Core\\BackgroundJobs\\GenerateMetadataJob job with ID 119428","userAgent":"--","version":"28.0.1.1","data":{"app":"cron"}}
{"reqId":"EgNErFpZbZCmFkP57K24","level":3,"time":"2024-01-14T15:47:26+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20814848728 bytes) at /var/www/nextcloud/lib/private/Files/Storage/Local.php#327","userAgent":"--","version":"28.0.1.1","data":{"app":"PHP"}}

Additional info

No response

Waldmeisda commented 6 months ago

sudo -u www-data php -d memory_limit=1G occ background-job:execute 119428 --force-execute
Job class:            OC\Core\BackgroundJobs\GenerateMetadataJob
Arguments:            null
Type:                 timed

Last checked:         2024-01-15T11:00:06+00:00
Reserved at:          -
Last executed:        2024-01-14T15:45:09+00:00
Last duration:        3871
Next execution:       2024-01-15T15:45:09+00:00

Forcing execution of the job

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20814848728 bytes) in /var/www/owncloud/lib/private/Files/Storage/Local.php on line 327

My idea was correct, its this one causing the issue.

solracsf commented 6 months ago

Does occ files:scan --generate-metadata -v triggers the same error? (maybe disable cronjob before executing the command)

Waldmeisda commented 6 months ago

I'll have a look,

I started it already according to https://github.com/nextcloud/server/issues/42308, but without the -v option. It returns a lot of different exif related errors, but nothing concerning so far.

Error during scan: exif_read_data(): Potentially invalid endianess, trying again with different endianness before imminent failure.
Error during scan: exif_read_data(): File not supported
Error during scan: exif_read_data(): Illegal IFD size

I'll start a verbose run afterwards.

Waldmeisda commented 6 months ago

Found the issue, the apple nextcloud client created and uploaded 20GB jpg .... grafik

Dont ask ...

nextcloud / server