[Bug]: PHP OPcache module is not properly configured: OPcache interned strings buffer is nearly full

LouisOuellet commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

The PHP OPcache module is not properly configured. See the documentation ↗ for more information. The OPcache buffer is nearly full. To assure that all scripts can be hold in cache, it is recommended to apply opcache.memory_consumption to your PHP configuration with a value higher than 1024.

So I am running version 25.0.5. Been encountering this error for a while. Kept increasing the limit but it always comes back. Even pushed the limit to 10gb, still would return the error. There seems to be something wrong with OPcache and Nextcloud Hub 25.

Note that I have several large network shares connected using external storage.

Steps to reproduce

Install a LAMP server with PHP 8.1 and all the required module
Add the recommended OPcache settings to php.ini
Login on Nextcloud and go to Administration - Overview

Expected behavior

Cache should regenerate according to demand.

Installation method

Community Manual installation with Archive

Nextcloud Server version

25

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[X] Default user-backend (database)
[X] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.albcie.com",
            "cloud.albgroupe.com",
            "cloud.albcustoms.com",
            "cloud.alblogistique.com",
            "192.168.20.253"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "http:\/\/cloud.albcie.com",
        "dbtype": "mysql",
        "version": "25.0.5.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": false,
        "logtimezone": "GMT-5",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "log_type": "owncloud",
        "logfile": "\/var\/www\/cloud.albcie.com\/log\/nextcloud.log",
        "loglevel": "2",
        "logdateformat": "F d, Y H:i:s",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "default_phone_region": "CA",
        "allow_local_remote_servers": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mysql.utf8mb4": true,
        "simpleSignUpLink.shown": false,
        "app_install_overwrite": [
            "keeweb",
            "nextant",
            "uploaddetails",
            "documentserver_community",
            "passman",
            "apporder",
            "drawio",
            "user_external",
            "limit_login_to_ip"
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis"
    }
}

List of activated Apps

Enabled:
  - apporder: 0.15.0
  - bookmarks: 12.1.0
  - bruteforcesettings: 2.5.0
  - calendar: 4.3.1
  - camerarawpreviews: 0.8.1
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contacts: 5.2.0
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - documentserver_community: 0.1.13
  - drawio: 2.1.0
  - external: 5.0.2
  - externalportal: 1.1.0
  - federatedfilesharing: 1.15.0
  - files: 1.20.1
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_retention: 1.14.1
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_texteditor: 2.15.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - groupfolders: 13.1.1
  - groupquota: 0.1.10
  - integration_github: 1.0.15
  - keeweb: 0.6.12
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - notes: 4.7.2
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - onlyoffice: 7.8.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - polls: 4.1.8
  - previewgenerator: 5.2.2
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - quota_warning: 1.16.0
  - related_resources: 1.0.4
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - smb_test: 0.3.6
  - spreed: 15.0.5
  - suspicious_login: 4.3.0
  - systemtags: 1.15.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.1
  - theming_customcss: 1.13.0
  - twofactor_backupcodes: 1.14.0
  - user_external: 3.1.0
  - user_ldap: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
Disabled:
  - activity: 2.6.1
  - admin_audit: 1.8.0
  - announcementcenter: 6.5.1
  - appointments: 1.14.13
  - audioplayer: 3.3.1
  - browser_warning: 1.0.0
  - checksum: 1.2.1
  - emlviewer: 1.0.7
  - encryption
  - epubreader: 1.4.7
  - event_update_notification: 2.1.0
  - extract: 1.3.5
  - federation: 1.8.0
  - files_accesscontrol: 1.15.1
  - files_automatedtagging: 1.15.3
  - files_frommail: 0.4.0
  - files_fulltextsearch: 25.0.0
  - files_fulltextsearch_tesseract: 25.0.0
  - files_trackdownloads: 1.7.0
  - firstrunwizard: 2.2.1
  - forms: 3.2.0
  - fulltextsearch: 25.0.0
  - impersonate: 1.12.0
  - integration_mattermost: 1.0.3
  - limit_login_to_ip: 3.1.0
  - mail: 2.2.5
  - nextant: 1.0.10
  - nextcloud_announcements: 1.7.0
  - ocr: 6.0.58
  - passman: 2.4.4
  - phonetrack: 0.7.4
  - radio: 0.6.6
  - rainloop: 7.2.6
  - ransomware_detection: 0.10.0
  - recommendations: 0.6.0
  - serverinfo: 1.8.0
  - socialsharing_email: 2.5.0
  - support: 1.0.1
  - survey_client: 1.5.0
  - talk_matterbridge: 1.26.0
  - twofactor_totp
  - updatenotification: 1.8.0
  - w2g2: 3.3.9
  - workflow_pdf_converter: 1.10.1

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

too large

Additional info

No response

szaimen commented 1 year ago

cc @MichaIng

MichaIng commented 1 year ago

Please try the following:

Remove any OPcache size related PHP ini setting
Increase the OPcache memory consumption until Nextcloud is happy. For a single Nextcloud instance, this should be already below the default value (128 MiB), i.e. no change needed, since Nextcloud alone isn't able to use more than 64 MiB.
Then set the interned strings buffer size to 25% of that (32 MiB of OPcache memory consumption is not changed), which should mute the recommendation even if it is fully used.

For still unknown reasons, some web applications (I observe the same on one Wordpress instance) and in case of Nextcloud some apps or combination of apps or usage cases, fill the interned strings buffer without limit. It might be an OPcache issue if e.g. no proper garbage collection is done or strings from cached PHP scripts are not evicted once the script itself is. I don't know a way to investigate since there is no API at all for the interned strings buffer to e.g. check the content or evict strings.

However, in affected cases this could have led to a cycle of:

Admin increases interned strings buffer as of recommendation
As interned strings buffer immediately uses overall OPcache memory, this leads to another recommendation about the OPcache memory consumption value, which admin then might raise, too.
Admin again increases interned strings buffer etc

So with a recent Nextcloud version we changed the check to only show an interned strings buffer recommendation if it is below 25% of the overall OPcache size, otherwise we assume that it just be filled without limit. On usual Nextcloud instances btw the interned strings buffer usage is somewhere around 4 MiB, 2.72 MiB currently on my personal instance, so 25% of 128 MiB = 32 MiB is already far above the expected need.

LouisOuellet commented 1 year ago

Here are my php settings for the nextcloud instance:

memory_limit=4G
upload_max_filesize = 32G
post_max_size = 32G
upload_tmp_dir = /var/www/cloud.albgroupe.com/home/albmanagercloud/tmp/
opcache.enable=1
opcache.interned_strings_buffer=256
opcache.max_accelerated_files=10000
opcache.memory_consumption=1024
opcache.save_comments=1
opcache.validate_timestamps=0
disable_functions =

Note, I am using ISPConfig as a web panel to manage all web services. So these settings are only for the nextcloud site. I tried increasing both opcache.interned_strings_buffer and opcache.memory_consumption to 10x the above limit and still have the error.

The server I am using has 128gb of RAM and is using a RAID10 array of SSDs for storage. Increasing the limit to 10gb is not a problem. But I also doubt Nextcloud needs that much either.

MichaIng commented 1 year ago

Again, please try to:

Remove opcache.interned_strings_buffer, opcache.max_accelerated_files and opcache.memory_consumption, restart PHP
Use Nextcloud for a while (loop through all apps' pages) then check for (first only) a opcache.memory_consumption a recommendation. You should not it if this is for a single Nextcloud instance only, as it uses 20-50 MiB and not the 1024 MiB you applied. However, if you do see it (OPcache shared among multiple instances/web applications), then raise opcache.memory_consumption until the recommendation is gone. Restart PHP after each change!
After that, set opcache.interned_strings_buffer to 25% of opcache.memory_consumption and restart PHP.

If afterwards the warning still shows up, that settings are not applied correctly. You can verify this e.g. via https://github.com/amnuts/opcache-gui: Put it into your web dir and access it directly to see actual OPcache settings and usage.

LouisOuellet commented 1 year ago

Ok I removed all the opcache settings and it not giving any errors about opcache anymore. Is there a way to confirm that nextcloud can still use it though? and that it is using it?

MichaIng commented 1 year ago

It is not Nextcloud which uses it but this is all handled automatically PHP-internally. If OPcache is enabled (which is by default since PHP 7.0), PHP automatically caches the opcode of all loaded PHP scripts up to the limits, including "interned strings". You can check cache usage and a list of all cached scripts with https://github.com/amnuts/opcache-gui.

Since scripts are cached on load/demand, the recommendation (it is not an "error"!) may show up at a later time.

LouisOuellet commented 1 year ago

Ok I will continue to monitor it.

nextcloud / server