cron: Long running jobs - Memory & cpu footprint will increase continously

Githopp192 commented 1 month ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

cron job is running 6h 20 minutes (now, CET Time 10:22 / am)

1528 4-18:40:27 /usr/sbin/crond -n 2176627 06:20:58 /bin/sh -c /bin/php -f /var/www/html/nextcloud/cron.php &> /dev/null 2176628 06:20:58 /bin/php -f /var/www/html/nextcloud/cron.php

may be this has to do with the config.php parameter:

'maintenance_window_start' => 2,

Steps to reproduce

monitor cron
cron consumes a lot of cpu and memory
after restarting apache, php-fpm, redis - memory & cpu footprint normalize

Expected behavior

cron job should not take tool long to do the maintenance cron job should not increase memory & cpu footprint continously, that the server gets in trouble

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

29

Operating system

RHEL/CentOS

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 28 to 29)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

Nextcloud version:29.0.4.1
Red Hat Enterprise Linux release 8.10 (Ootpa)
10.6.18-MariaDB,
Apache/2.4.37
PHP 8.3.10

******************

{
    "system": {
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": ***REMOVED SENSITIVE VALUE***,
            "dbindex": ***REMOVED SENSITIVE VALUE***,
            "timeout": 1.5,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "overwriteprotocol": "https",
        "dbtype": "mysql",
        "version": "29.0.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "debug": false,
        "auth.bruteforce.protection.enabled": true,
        "check_for_working_htaccess": true,
        "updater.release.channel": "stable",
        "upgrade.disable-web": false,
        "data-fingerprint": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/***REMOVED SENSITIVE VALUE***\/log\/nextcloud.log",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "***REMOVED SENSITIVE VALUE***",
        "session_lifetime": 1200,
        "session_keepalive": false,
        "logtimezone": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "***REMOVED SENSITIVE VALUE***",
        "knowledgebaseenabled": false,
        "log_rotate_size": 3145728,
        "mail_sendmailmode": "smtp",
        "app_install_overwrite": [
            "passman",
            "dicomviewer",
            "radio",
            "cookbook",
            "health",
            "issuetemplate",
            "documentserver_community",
            "imageconverter",
            "epubreader",
            "workflow_pdf_converter",
            "signlive",
            "apporder",
            "rainloop",
            "integration_restya"
        ],
        "maintenance_window_start": 2
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - analytics: 4.14.0
  - announcementcenter: 6.8.1
  - audioplayer: 3.4.1
  - bookmarks: 14.2.3
  - bruteforcesettings: 2.9.0
  - cadviewer: 9.71.1
  - calendar: 4.7.15
  - calendar_resource_management: 0.8.0
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - collectives: 2.13.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - cookbook: 0.11.1
  - cospend: 1.6.1
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.1
  - dicomviewer: 2.1.2
  - drawio: 3.0.2
  - drop_account: 2.6.0
  - electronicsignatures: 3.0.2
  - end_to_end_encryption: 1.15.2
  - event_update_notification: 2.4.0
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_antivirus: 5.5.7
  - files_automatedtagging: 1.19.0
  - files_downloadactivity: 1.17.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_pdfviewer: 2.10.0
  - files_photospheres: 1.29.1
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - flow_notifications: 1.9.0
  - forms: 4.2.4
  - geoblocker: 0.5.14
  - gpxpod: 5.0.18
  - groupfolders: 17.0.1
  - health: 2.2.2
  - imageconverter: 2.0.2
  - impersonate: 1.16.0
  - integration_collaboard: 1.0.8
  - integration_deepl: 1.2.0
  - integration_discourse: 2.0.7
  - integration_dropbox: 3.0.0
  - integration_github: 2.0.7
  - integration_gitlab: 3.0.1
  - integration_google: 3.0.0
  - integration_jira: 1.2.0
  - integration_mastodon: 2.0.6
  - integration_openai: 2.0.3
  - integration_openproject: 2.6.3
  - integration_openstreetmap: 1.0.12
  - integration_reddit: 2.0.4
  - integration_replicate: 3.0.0
  - integration_zammad: 2.1.0
  - keeweb: 0.6.19
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.6
  - maps: 1.4.0
  - money: 0.28.0
  - nextcloud_announcements: 1.18.0
  - nextpod: 0.7.5
  - notes: 4.10.1
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - passman: 2.4.9
  - password_policy: 1.19.0
  - passwords: 2024.7.20
  - photos: 2.5.0
  - polls: 7.2.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - quicknotes: 0.8.23
  - quota_warning: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.4
  - richdocumentscode: 24.4.502
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - side_menu: 3.13.1
  - signlive: 0.9.5
  - sketch_picker: 1.0.2
  - spreed: 19.0.7
  - support: 1.12.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - tables: 0.7.5
  - tasks: 0.16.0
  - terms_of_service: 2.5.0
  - text: 3.10.1
  - theming: 2.4.0
  - timemanager: 0.3.15
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - welcome: 1.2.0
  - workflow_media_converter: 1.11.7
  - workflow_pdf_converter: 1.14.0
  - workflow_script: 1.14.1
  - workflowengine: 2.11.0
  - yumisign_nextcloud: 1.29.2
Disabled:
  - admin_audit: 1.19.0 (installed 1.6.0)
  - audioplayer_editor: 0.3.0 (installed 0.3.0)
  - audioplayer_sonos: 1.3.0 (installed 1.3.0)
  - breezedark: 29.0.0 (installed 29.0.0)
  - cms_pico: 1.1.0-beta.1 (installed 1.0.20)
  - data_request: 1.15.0 (installed 1.15.0)
  - encryption: 2.17.0
  - epubreader: 1.4.7
  - external: 5.4.0 (installed 5.4.0)
  - extract: 1.3.6 (installed 1.3.6)
  - files_3d: 0.5.0 (installed 0.5.0)
  - files_accesscontrol: 1.19.1 (installed 1.19.1)
  - files_external_onedrive: 1.1.0 (installed 1.1.0)
  - files_fulltextsearch: 29.0.0 (installed 29.0.0)
  - files_markdown: 2.4.1 (installed 2.4.1)
  - files_mindmap: 0.0.30 (installed 0.0.30)
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - flowupload: 1.1.3 (installed 1.1.3)
  - fulltextsearch: 29.0.0 (installed 29.0.0)
  - fulltextsearch_elasticsearch: 29.0.1 (installed 29.0.1)
  - gdatavaas: 29.2.2 (installed 29.2.2)
  - gpoddersync: 3.9.0 (installed 3.9.0)
  - gpxedit: 0.0.14 (installed 0.0.14)
  - gpxmotion: 0.1.0 (installed 0.1.0)
  - hibp: 0.1.0 (installed 0.1.0)
  - integration_homeassistant: 0.0.5 (installed 0.0.5)
  - integration_onedrive: 3.2.2 (installed 3.2.2)
  - integration_pexip: 1.0.4 (installed 1.0.4)
  - integration_twitter: 1.0.7 (installed 1.0.7)
  - integration_whiteboard: 0.0.14 (installed 0.0.14)
  - issuetemplate: 0.7.0 (installed 0.7.0)
  - joplin: 0.0.25 (installed 0.0.25)
  - libresign: 9.2.3 (installed 9.2.3)
  - news: 24.0.0 (installed 24.0.0)
  - ojsxc: 5.0.0 (installed 5.0.0)
  - rainloop: 7.2.6 (installed 7.2.6)
  - ransomware_protection: 1.14.0 (installed 1.14.0)
  - registration: 2.4.0 (installed 2.4.0)
  - survey_client: 1.17.0 (installed 1.3.0)
  - talk_matterbridge: 1.26.0 (installed 1.26.0)
  - talked: 0.5.0 (installed 0.5.0)
  - tuya_cloud: 0.0.4 (installed 0.0.4)
  - user_ldap: 1.20.0
  - video_converter: 1.0.6 (installed 1.0.6)
  - weather: 1.7.7 (installed 1.7.7)
  - workflow_ocr: 1.29.0 (installed 1.29.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

together with the observation of cron, it also shows, among other things, that the php-fpm process uses a lot of space in /tmp/

du -hs /tmp/systemd-private-4c513a85a5cb462b92e805310c385d9e-php-fpm.service-jkmBSp/tmp/coolwsd.0u42WDiYQI/jails/* 5.1G /tmp/systemd-private-4c513a85a5cb462b92e805310c385d9e-php-fpm.service-jkmBSp/tmp/coolwsd.0u42WDiYQI/jails/1976183-9bea7ef2

space used coming from Nextcloud Office (collabora/coolwsd). Space consumption is growing steadily up to 50..60..70G in /tmp

May be the "cron behave" has something to do with this ? (side effect) ?

https://github.com/nextcloud/richdocuments/issues/2676 https://github.com/CollaboraOnline/richdocumentscode/issues/176

ChristophWurst commented 1 month ago

Logs of https://github.com/nextcloud/server/pull/45804 and https://github.com/nextcloud/server/pull/42391 would be helpful to identify the job and its arguments.

Githopp192 commented 1 month ago

cron job did end now, run time ~ 7 - 8 hours

memory & cpu consumption did normalize for that moment - no increase in ==> /tmp/php-fpm

not sure if the new maintenance feature could be the culprit - did change the value now from:

'maintenance_window_start' => 2,

to :

'maintenance_window_start' => 100, (disabling it).

Do you need any other logs, when yes - which ones ?

ChristophWurst commented 1 month ago

Do you need any other logs, when yes - which ones ?

The messages of of https://github.com/nextcloud/server/pull/45804 and https://github.com/nextcloud/server/pull/42391.

Githopp192 commented 1 month ago

Do you need any other logs, when yes - which ones ?

The messages of of #45804 and #42391.

you mean the nextcloud.log ?

joshtrichards commented 1 month ago

you mean the nextcloud.log ?

Yes. The same one we ask for in our bug reporting template. :)

Githopp192 commented 1 month ago

you mean the nextcloud.log ?

Yes. The same one we ask for in our bug reporting template. :)

Yes, i'd love to do that, but as there is no possibility (afik), to attach this as "non public" .. I would say refrain from putting this log with very sensitive user information (IP...and more) to public

joshtrichards commented 1 month ago

We only need a sample of the long running job entries. We don't need IPs or any identifying information. And we certainly don't want the entire log file.

Without the log entry I'm not sure what you expect us to do. :-)

You can also look at it yourself though: it should indicate what job and what app.

Githopp192 commented 1 month ago

Yes, Josh - can understand :-)

I'll see, what i can do - related to filter out the log ...

on the other hand .. is not only an issue, which affects my server - this issue could become a critical one for the enterprise ==>

https://github.com/CollaboraOnline/richdocumentscode/issues/187#issuecomment-2276750318

it seems to be a cron - collabora side effect

ChristophWurst commented 1 month ago

it seems to be a cron - collabora side effect

Do you have a log line to back this theory?

Githopp192 commented 1 month ago

it seems to be a cron - collabora side effect

Do you have a log line to back this theory?

no - i haven't - but since i disabled the maintenance specifc run ('maintenance_window_start' => 100, (disabling it), the issue did not re-occur !?

(issue = long running cron & increasing /tmp/php stuff)

Githopp192 commented 4 weeks ago

you could check the comment of @cableTh0rn in ==>

https://github.com/CollaboraOnline/richdocumentscode/issues/187

"I run a check script to log the time tmp files gets bigger. It is just the maintenance_window time, 100GB in just one hour"

may be the cron maintenance window ('maintenance_window_start') has an influence on the behavior of collabora ?

In this case it would be worth not to close the case, but to investigate it, because there are obviously some people with the same issue ? :-)

ChristophWurst commented 4 weeks ago

I'm happy to reopen once we have some logs that clearly indicate where the problem lies. Right now we have symptoms and speculation. We don't know why or where the memory grows.

nextcloud / server