[Bug]: OPcache buffer & interned strings buffers always full

idressos commented 4 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

I have two Nextcloud installations, on different dedicated servers managed by Virtualmin. One installation is fresh and the other one has received upgrades.

On both installations, in the Administration Overview page, I get the following warning:

The PHP OPcache module is not properly configured. The OPcache buffer is nearly full. To assure that all scripts can be hold in cache, it is recommended to apply "opcache.memory_consumption" to your PHP configuration with a value higher than "512". The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply "opcache.interned_strings_buffer" to your PHP configuration with a value higher than "32"..

No matter what value I set the memory_consumption and interned_strings_buffer options to, even insanely high values, the warning will still show, still prompting me to raise the values to something even higher.

This started happening on Nextcloud versions >=28.

Steps to reproduce

Start with a Nextcloud installation version >= 28

Expected behavior

No warnings concerning OPcache buffers.

Installation method

Community Manual installation with Archive

Nextcloud Server version

master

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.aviettechnic.org"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.3.2",
        "overwrite.cli.url": "https:\/\/cloud.aviettechnic.org",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "default_phone_region": "MT",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "defaultapp": "files",
        "maintenance": false,
        "loglevel": 3,
        "data-fingerprint": "c4fd14e1fb3af2acb9bd253d3cb2e2aa",
        "maintenance_window_start": 1
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - admin_audit: 1.18.0
  - analytics: 4.12.0
  - announcementcenter: 6.8.0
  - appointments: 2.0.4
  - approval: 1.2.0
  - audioplayer: 3.4.1
  - calendar: 4.6.7
  - checksum: 1.2.3
  - circles: 28.0.0-dev
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contactsinteraction: 1.9.0
  - dashboard: 7.8.0
  - dav: 1.29.1
  - deck: 1.12.2
  - external: 5.3.1
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_accesscontrol: 1.18.0
  - files_antivirus: 5.4.2
  - files_automatedtagging: 1.18.0
  - files_bpm: 1.3.0
  - files_confidential: 2.1.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - firstrunwizard: 2.17.0
  - forms: 4.1.1
  - groupfolders: 16.0.4
  - impersonate: 1.15.0
  - integration_openai: 1.2.1
  - llm: 1.2.1
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.2
  - notifications: 2.16.0
  - notify_push: 0.6.10
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - quota_warning: 1.18.0
  - recognize: 6.1.1
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - richdocuments: 8.3.2
  - richdocumentscode: 23.5.904
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - side_menu: 3.11.8
  - support: 1.11.0
  - suspicious_login: 6.0.0
  - systemtags: 1.18.0
  - tasks: 0.15.0
  - text: 3.9.1
  - theming: 2.3.0
  - translate: 2.1.0
  - twofactor_backupcodes: 1.17.0
  - updatenotification: 1.18.0
  - user_status: 1.8.1
  - viewer: 2.2.0
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - bruteforcesettings: 2.8.0
  - encryption: 2.16.0
  - files_external: 1.20.0
  - survey_client: 1.16.0 (installed 1.16.0)
  - twofactor_totp: 10.0.0-beta.2
  - user_ldap: 1.19.0

Nextcloud Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- core
    - INVALID_HASH
        - core/js/mimetypelist.js

Raw output
==========
Array
(
    [core] => Array
        (
            [INVALID_HASH] => Array
                (
                    [core/js/mimetypelist.js] => Array
                        (
                            [expected] => 550ab566d30693bfa24ec4b15d9df87731ae8a3be8f79dabf94757e5b8b20eec6e4b678f17af1718297f2872f6b04519eeb024d1dff11947f29da431c7f11201
                            [current] => 24e6faa4a1361e51ff70027024f41c4b37fdb5f6f347ceb095450ddbde2d0a6b6dd1403be92a912d2889064a9736a9bed89d7dc62470035fc1299d4b977c5f46
                        )

                )

        )

)

Nextcloud Logs

No response

Additional info

I mentioned above this affects two Nextcloud instances. The config report and app list concern the freshly installed one.
The invalid hash of file core/js/mimetypelist.js is another issue I am facing, and I've opened a separate issue.

PHP-FPM config:

[1706092746572748]
listen.mode = 0660
listen = /run/php/1706092746572748.sock
pm = dynamic
pm.max_children = 16
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 8
php_value[upload_tmp_dir] = [REDACTED]
php_value[session.save_path] = [REDACTED]
php_value[log_errors] = On
php_admin_value[post_max_size] = 16G
php_admin_value[upload_max_filesize] = 16G
php_admin_value[max_execution_time] = 3600
php_admin_value[max_input_time] = 3600
php_admin_value[memory_limit] = 8G
php_admin_value[opcache.enable] = 1
php_admin_value[opcache.memory_consumption] = 512
php_admin_value[opcache.interned_strings_buffer] = 32
php_admin_value[opcache.max_accelerated_files] = 10000
php_admin_value[opcache.revalidate_freq] = 60
php_admin_value[opcache.validate_timestamps] = 0
php_admin_value[opcache.save_comments] = 1
php_admin_value[opcache.jit] = 1255
php_admin_value[opcache.jit_buffer_size] = 128M

idressos commented 4 months ago

@Matthias-Ab

joshtrichards commented 4 months ago

I have two Nextcloud installations, on different dedicated servers managed by Virtualmin. One installation is fresh and the other one has received upgrades. On both installations, in the Administration Overview page, I get the following warning:

Do both have similar app combinations? Are both using dedicated FPM pools for Nextcloud or sharing with other apps?

You might also find this of interest to see what's going on: https://github.com/amnuts/opcache-gui

The checks you're hitting are these:

https://github.com/nextcloud/server/blob/c451829579de14fbae33b9f59075a025471066c6/apps/settings/lib/SetupChecks/PhpOpcacheSetup.php#L98-L103

https://github.com/nextcloud/server/blob/c451829579de14fbae33b9f59075a025471066c6/apps/settings/lib/SetupChecks/PhpOpcacheSetup.php#L105-L115

idressos commented 4 months ago

Both instances have almost identical app combinations. I used OPcache-GUI and it looks like the buffers actually are full. Specifically the interned strings buffer is always 100% full, and general memory consumption is at >=85% depending on how high the limit is. The more memory I allocate, the more the usage is.

a-bern commented 4 months ago

I faced the same issue with my setup. It looks like the interned_strings_buffer cant be modified using php_admin_value. Although phpinfo shows opcache.interned_strings_buffer as increased, the opcache module is still using the default value.

I saw that with the mentioned opcache-gui tool.

Increasing the opcache.interned_strings_buffer value in the global /etc/php/8.2/fpm/conf.d/10-opcache.ini or /etc/php/8.2/fpm/php.ini did the trick for me.

cat /etc/php/8.2/fpm/conf.d/10-opcache.ini
; configuration for php opcache module
; priority=10
zend_extension=opcache.so
opcache.jit=off
opcache.interned_strings_buffer=32

a-bern commented 4 months ago

Maybe the nextcloud check could include a check that verifies if the php setting "interned_strings_buffer" actually matches the interned strings buffer reported by the opcache module.

I see a lot of issues / discussions around this error. Including the above check would perhaps help a lot of people.

Might also be related to this upstream issue https://github.com/php/php-src/issues/11715

rainerstudhalter commented 1 month ago

Increasing the opcache.interned_strings_buffer value in the global /etc/php/8.2/fpm/conf.d/10-opcache.ini or /etc/php/8.2/fpm/php.ini did the trick for me.

I can confirm this and i also had to set memory_consumption in the ini. Although I recommend putting those values into a custom ini:

cat /etc/php/8.3/fpm/conf.d/99-custom.ini 
[opcache]
opcache.interned_strings_buffer=64
opcache.memory_consumption=2048

nextcloud / server