search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.67k stars 4k forks source link

[Bug]: The OPcache interned strings buffer is nearly full #31223

Closed AndyXheli closed 2 years ago

AndyXheli commented 2 years ago

⚠️ This issue respects the following points: ⚠️

Bug description

The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 8.

image

Steps to reproduce

  1. Server Overview

Expected behavior

NA

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "1.1.1.14",
            "cloud.next.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "23.0.2.1",
        "overwrite.cli.url": "https:\/\/cloud.next.com",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "simpleSignUpLink.shown": false,
        "trashbin_retention_obligation": "auto,30",
        "versions_retention_obligation": "auto,1",
        "skeletondirectory": "",
        "default_phone_region": "US",
        "allow_local_remote_servers": true,
        "mail_smtpmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_smtpport": "587",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "maintenance": false,
        "updater.release.channel": "stable",
        "theme": "",
        "loglevel": 2,
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - admin_audit: 1.13.0
  - bruteforcesettings: 2.3.0
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contactsinteraction: 1.4.0
  - dashboard: 7.3.0
  - dav: 1.21.0
  - federatedfilesharing: 1.13.0
  - federation: 1.13.0
  - files: 1.18.0
  - files_external: 1.15.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - groupfolders: 11.1.2
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - nextcloud_announcements: 1.12.0
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - officeonline: 1.1.2
  - password_policy: 1.13.0
  - photos: 1.5.0
  - previewgenerator: 4.0.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - ransomware_protection: 1.12.0
  - recommendations: 1.2.0
  - sendent: 1.2.10
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - socialsharing_email: 2.4.0
  - spreed: 13.0.3
  - support: 1.6.0
  - survey_client: 1.11.0
  - suspicious_login: 4.1.0
  - systemtags: 1.13.0
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_admin: 3.2.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_totp: 6.2.0
  - updatenotification: 1.13.0
  - user_status: 1.3.1
  - viewer: 1.7.0
  - weather_status: 1.3.0
  - workflowengine: 2.5.0
Disabled:
  - circles: 22.1.1
  - encryption
  - user_ldap

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

.

szaimen commented 2 years ago

cc @MichaIng

AndyXheli commented 2 years ago

I did a sudo service apache2 restart and it went away but then comes back after a while

MichaIng commented 2 years ago

@andyxheli Why do you think this is a bug, or was it just the best issue template you found? This shows up when the available OPcache interned strings buffer size is more than 90% filled. You have a relatively long list of enabled apps, in which case this seems to be not too unexpected. We had one more such report, and another one where Nextcloud was not the only web application running on the same PHP (and hence OPcache) pool.

On Debian with PHP8.0, I suggest to raise it with an own override config module, doubling it to 16 MiB:

# Create an own PHP module with highest priority
cat << '_EOF_' > /etc/php/8.0/mods-available/local-opcache.ini
; Raise OPcache interned strings buffer for Nextcloud
; priority=99
opcache.interned_strings_buffer=16
_EOF_
# Enable the module
phpenmod local-opcache
# Restart Apache2 to apply the new module
systemctl restart apache2

I did a sudo service apache2 restart and it went away but then comes back after a while

This makes sense since the OPcache as well as its interned strings buffer are filled on demand by default, i.e. each script is loaded once requested the first time. When you restart Apache2, the OPcache instance is destroyed and it the buffer is filled while you using the Nextcloud web interface and the individual apps.

Since this seems to be not as rare as I expected, it makes sense that we add this explicitly to the docs: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache Also I see a link to the docs from the admin panel is missing. I'll add this as well now that docs and admin panel recommendations match.

Not sure whether it breaks the scope of the docs, but I could also add the steps like above to add a config override, the IMHO preferred way since no package files are touched, so that changes can be reverted easily and package manager upgrades can update the default configuration files without interactive confirmation and loosing customisations. In case, does anyone know whether phpenmod/phpdismod is specific to Debian/Ubuntu/APT or whether these commands are available on other distros, RPM and AUR based ones, and upstream PHP builds as well?

AndyXheli commented 2 years ago

@MichaIng Sorry I searched everywhere for something regarding this and nothing came up for me I even check the docs and I didn't see anything about a recommended opcache.interned_strings_buffer=16. so I though to submit a bug. I should change my settings from 8 to 16 ?

MichaIng commented 2 years ago

Okay, no problem, I asked just in case it showed something wrong, e.g. you raised the value already or verified that in fact the buffer is not so full.

I didn't see anything about a recommended opcache.interned_strings_buffer=16.

And this makes sense as it depends on the enabled apps and in case other PHP applications running on the same PHP/OPcache instance. In most cases, the default 8 MiB are sufficient, in rare cases, as far we could see, 16 MiB are required to keep everything stored in cache. But it wouldn't make sense to recommend this to everyone when the additional 8 MiB reserved in total OPcache was just empty in most cases 😉. So the warnings do now show up based on actual current cache usage.

But as I said, it makes sense to mention it in the docs and link that docs section from within the admin panel warnings. I'll raise a PR the next days.

I should change my settings from 8 to 16?

This I would recommend, using e.g. the commands I provided above. Note that this is not mandatory, it just assures that all strings present in Nextcloud's PHP scripts stay cached and hence speeds up access performance a bit. The interned strings buffer is quite a clever implementation which stores repeating strings across all cached scripts just a single time in shared memory and additionally persistent for future requests (which doesn't seem to be the default according to some readings).

toxpal commented 2 years ago

All my NC installations were working fine for years, but right after upgrading to 23.0.2 I get these errors on every single installation.

ghost commented 2 years ago

I have the same problem, wenn i raise it to 16 it sayy raise it higher, wenn i raise it to 32 it says raise it higher

toxpal commented 2 years ago

I have the same problem, wenn i raise it to 16 it sayy raise it higher, wenn i raise it to 32 it says raise it higher

Exactly the same. No matter how high I set new limits, error re-appears again and again.

MichaIng commented 2 years ago

All my NC installations were working fine for years

And it still does. Note that the admin panel warnings/recommendations are nothing mandatory, just a hint what may be improved according to how far Nextcloud is able to check it. But warnings may be wrong or not apply in a particular case, e.g. I keep ignoring warnings about missing imagick (I do not, and do not want to use ImageMagic and Nextcloud works very well with GD), missing bcmath and gmp (I do not use WebAuthn, and those are not required for anything else) and about the missing X-Download-Options header (I do not use Internet Explorer, nor does any other user of my instance). I however understand that one usually wants to have the admin panel free of warnings, which is why I was already requesting to solve the bcmath and gmp recommendations differently, embedding them right into the WebAuthn settings, keep observing the discussion about getting rid of imagick (which is known to have potential security issues with SVG embedded code) and cannot await #20855 to be fully done.

Back to topic: I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB, so @copiis @toxpal are you running only a single Nextcloud instance on that webserver/PHP pool or are there other web applications (which do use PHP)? If it is really just one Nextcloud, can you show your occ app:list, please?

fantasya-pbem commented 2 years ago

I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB

My little Nextcloud has the same issue, too. I used opcache before, but immediately after upgrade from 22 to 23 a couple of days ago I got this opcache warning, too. I tried to get rid of it by doubling interned_strings_buffer from 8 to 16 and now 32, but this warning still exists. It feels like a newly introduced bug. Is there a possibility to roughly estimate the needed buffer space NC needs?

MichaIng commented 2 years ago

Same as for the others, if Nextcloud is really the only website/application running on that webserver and PHP instance, please post the output of occ app:list so we may find a similarity between your apps which might cause this.

Is there a possibility to roughly estimate the needed buffer space NC needs?

The admin panel does exactly this. It looks how much of the available interned strings buffer is in fact used, and when >90% of available size, it prints the recommendation. If Nextcloud is really the only website, then obviously it requires that much interned strings buffer in your particular case, at least when you want to assure that all strings are kept in cache.

fantasya-pbem commented 2 years ago

The admin panel does exactly this. It looks how much of the available interned strings buffer is in fact used, and when >90% of available size, it prints the recommendation.

So this check is a new NC 23 feature?

MichaIng commented 2 years ago

NC v23.0.1 to be precise.

ghost commented 2 years ago

Back to topic: I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB, so @copiis @toxpal are you running only a single Nextcloud instance on that webserver/PHP pool or are there other web applications (which do use PHP)? If it is really just one Nextcloud, can you show your occ app:list, please?

On the webserver is one Nextcloud an one Friendica instalation

MichaIng commented 2 years ago

an one Friendica instalation

Okay, in this case it is not so unexpected that the default is insufficient. Not sure how large Friendica is. You can btw monitor OPcache usage via opcache-gui which allows to browse and search through all cached scripts, see their size in cache etc. Sadly the interned strings buffer cannot be listed like that, and I'm not sure about any way to further analyse its content.

E.g. I have a server with phpBB, Wordpress, Matomo and some own pages and it uses ~150 MiB interned strings buffer (I set it to 256 MiB, giving 192 MiB effective space, always 75%) and up to 400 MiB overall OPcache size. My private Nextcloud instance uses 3.5 MiB/32 MiB, so the ratio between interned strings buffer and overall OPcache usage (the letter includes the first) is quite different depending on which application is used, and in Nextcloud obviously depending on used apps. In case of phpBB and Wordpress they both create large (up to 1 MiB) temporary PHP scripts as some own cache layer, which probably plays a role, especially when those are not evicted on removal (which would be basically a bug), but I haven't analysed it further for now.

AndyXheli commented 2 years ago

you also also use CLI https://github.com/gordalina/cachetool

MichaIng commented 2 years ago

Ah nice, I didn't know that one. Also good for the docs as alternative.

AndyXheli commented 2 years ago

Here a website that lists all the recommended monitoring options https://haydenjames.io/php-performance-opcache-control-panels/

AndyXheli commented 2 years ago

image

MichaIng commented 2 years ago

To not only see the tool's own PHP/OPcache instance, AFAIK such needs to be passed:

--fcgi=/run/php/php8.0-fpm.socket

But this is for PHP-FPM. In case Apache2 with mod_php is used, instead this may work:

--web-path=/path/to/your/document/root --web-url=https://my.domain.org
gmeks commented 2 years ago

The value of 8 also seems to be the default in docker images. And im getting the same warning running of nextcloud:apache

MichaIng commented 2 years ago

8 MiB is the PHP default: https://www.php.net/manual/opcache.configuration.php

masterwishx commented 2 years ago

image

Also have this warning after added redis cache to last version of server:

memcache.distributed' => '\OC\Memcache\Redis', 'memcache.locking' => '\OC\Memcache\Redis', 'redis' => [ 'host' => '192.168.0.199', 'port' => 6379, 'dbindex' => 1, 'password' => 'password', ],

MichaIng commented 2 years ago

Redis and transactional file locking is completely unrelated to OPcache and the interned strings buffer. Simply follow the recommendation if you want to max performance.

vgnmnky commented 2 years ago

I have this error as well. Unfortunately, I'm very new to Docker (partly jumping in as a learning exercise), and so I don't know much yet about where the settings for associated services are or where (host, various containers) I'm supposed to enter commands, and I'm finding it difficult to find out.

I have a stack with Nextcloud (23.0.1, from new), MariaDB and Nginx Proxy Manager. I fixed a few other errors, and it works, but please could someone give me an idea where to look to change the opcache settings.

Tried to get a list of installed apps too, as others have posted, but I don't seem to be able to use the occ command anywhere. I have spent a fair bit of time trouble-shooting Nextcloud and other containers, which is part of the learning experience, but though I'm not an IT or CLI noob, I'm finding the level of assumed knowledge to be above me, so some small pointers to help me on my way would be really appreciated.

MichaIng commented 2 years ago

There is no way (I'm aware of) to configure PHP settings with the Docker container. It needs to be changed by the maintainer here (which still follows the old static recommendation): https://github.com/nextcloud/docker/blob/master/Dockerfile-alpine.template#L81-L86

The all-in-one appliance switched to opcache.interned_strings_buffer=16 already to cover more cases. With the default 128 MiB overall OPcache size, this is no problem since this is much more than enough. All other settings can be removed, matching PHP defaults anyway, opcache.revalidate_freq=1 is a performance decrease, should be either left at 2 (default) or even much higher. Especially in a container, one cannot simply edit Nextcloud scripts and expect changes to be effective within a second. config.php can be edited, but it should be more than sufficient to have changes taking effect within 5 seconds or longer, to decrease the amount of timestamp validations PHP does.

Ah, here it is already: https://github.com/nextcloud/docker/pull/1702

masterwishx commented 2 years ago

Simply follow the recommendation if you want to max performance.

I'm using nextcloud Linux server docker for unraid, so it's warning Becouse of update? What exactly the recommendation you mean? It's good idea to use redis for home server + APCu?

MichaIng commented 2 years ago

As linked above, there is a change on the way for the Nextcloud Docker container, to get the interned strings buffer increased and hence the warning solved in very most cases.

It's good idea to use redis for home server + APCu?

Yes, Redis is mandatory for a stable Nextcloud IMO, APCu the fastest local cache.

masterwishx commented 2 years ago

Yes, Redis is mandatory for a stable Nextcloud IMO, APCu the fastest local cache.

Thanks, I also saw that nextcloud was more quicker after added redis for distribution and locking, befor was only APCu for local.

MichaIng commented 2 years ago

Note that the distributed cache is only relevant when you connect multiple Nextcloud instances to it. I'm not sure about the internal logic, in which cases the distributed cache is used, but it would make sense to do it for all non-sensitive objects which are likely or known to be the same on multiple Nextcloud instances, like avatar images of shared users or such. Redis is slower than APCu, so when you define the distributed cache to use Redis, Nextcloud will use a slower cache for mentioned cache objects. As long as you do not connect multiple Nextcloud instances to that cache, skip the distributed cache setting to let Nextcloud use the local cache only (APCu) instead and Redis only for transactional file locking.

masterwishx commented 2 years ago

As long as you do not connect multiple Nextcloud instances to that cache, skip the distributed cache setting to let Nextcloud use the local cache only (APCu) instead and Redis only for transactional file locking.

  1. Do you mean multple instances of nextcloud server right? For now im using only one nextcloud on unraid in home server , maybe will add one more on VPS later...

  2. i was using only:

'memcache.local' => '\OC\Memcache\APCu',

and now:

'memcache.local' => '\OC\Memcache\APCu', 'memcache.distributed' => '\OC\Memcache\Redis', 'memcache.locking' => '\OC\Memcache\Redis', 'redis' => array ( 'host' => '192.168.0.199', 'port' => 6379, 'dbindex' => 1, 'password' => 'password',

So you mean to left only local cache APCu and Redis for filelocking like this?

'memcache.local' => '\OC\Memcache\APCu', 'memcache.locking' => '\OC\Memcache\Redis', 'redis' => array ( 'host' => '192.168.0.199', 'port' => 6379, 'dbindex' => 1, 'password' => 'password',

devnull4242 commented 2 years ago

Sorry.

But can someone explain why the change from 8 to 16 is nessessary from Nextcloud 23.0.0 to 23.0.2 ? I think the installed apps can not be the reason. That is nonsense. It must be a bug or a feature in Nextcloud 23.0.2. Perhaps I have only read it over in this thread.

MichaIng commented 2 years ago

Please read this issue from the beginning, or: https://help.nextcloud.com/t/nextcloud-23-02-opcache-interned-strings-buffer/134007/5?u=michaing

=> Not a bug, no nonsense, no necessity

devnull4242 commented 2 years ago

Thanks.

rarealphacat commented 2 years ago

Increase opcache to 16 still seeing this error, though random upload freeze (indefinitely) does not appear anymore. Tried changing to 32 and still seeing this error.

MichaIng commented 2 years ago

Uploads are not affected by OPcache at all. If Nextcloud is the only web application served by your webserver/PHP instance, you might help with your apps list to identify apps which use that many interned strings. Otherwise keep increasing the buffer until the recommendation disappears (note that you may need to increase opcache.memory_consumption as well which holds the interned strings buffer, among cached PHP scripts), or until the increased memory consumption causes downsides elsewhere.

AndyXheli commented 2 years ago

That strange I changed mine to 16 a few weeks ago restated the web service I haven't had that warning come up again.🤞🏼

MichaIng commented 2 years ago

I had mine at 4 MiB for a long time without the warning. It highly depends on the used apps and in case other web applications served. That's the whole point why recommendations are now OPcache usage based instead of fixed values (like before) 😉.

LukaszSellmann commented 2 years ago

Version 23.0.2 confirmed , message displays minutes after apache restart.

opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=16

Screenshot 2022-03-03 at 08 53 36

MichaIng commented 2 years ago

@LukaszSellmann Note that your screenshots shows CLI values, not the ones from the PHP/OPcache instance which serves Nextcloud, see: https://github.com/nextcloud/server/issues/31223#issuecomment-1043230780

Failsafe to obtain the correct value is accessing this script via web server:

<?php echo '<pre>'; print_r(opcache_get_status(false));
AndyXheli commented 2 years ago

@MichaIng so warning is back again now stating The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 16. Do i need to go up to 32 now ?

Here a php cachetool.phar opcache:status --cli output image

MichaIng commented 2 years ago

Do i need to go up to 32 now ?

If you want to address and mute the recommendation, then yes.

Here a php cachetool.phar opcache:status --cli output

Please read through my posts above: Run that CLI properly to show the Nextcloud instance's PHP OPcache usage and not the new/isolated CLI tool's own PHP OPcache instance and usage. If unsure, use the web interface or PHP script I posted and access them via browser, which is assured to show the correct OPcache instance's stats. But actually it is not required: The recommendation is based on exactly the same API without any bug known, hence you can trust the recommendation when it says that the interned strings buffer is nearly full, then it is.

AndyXheli commented 2 years ago

@MichaIng Thank you very much! for the reply. I updated it to 32 lets see what happens.

hugalafutro commented 2 years ago

If you use linuxserver.io nextcloud docker image you can edit php-local.ini with a value of your choosing which will persist container recreation/restart as outlined @ https://github.com/linuxserver/docker-nextcloud/issues/235#issuecomment-1047064507 As I do not use official docker image I cannot comment on how it functions there.

I had to go all the way up to 32 before the warning disappeared (for now, as it disappeared on previous increments, but returned shortly).

slawa-dev commented 2 years ago

@MichaIng Thank you very much! for the reply. I updated it to 32 lets see what happens.

It will not stop. I kept doubling, now at 256. Message still appears after a while.

MichaIng commented 2 years ago

@slawa-dev It is only a single Nextcloud instance running on this server?

AndyXheli commented 2 years ago

@MichaIng i just setup a new server Nextcloud only running on it and got this message after a day.

The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 16. I'll go up to 32 but theres something here that causing this

MichaIng commented 2 years ago

Guys, there is no need to keep posting when you see this warning. There is no single evidence that it is wrong or a bug, but the actual OPcache stats are read correctly. So when you see the recommendation, then indeed your interned strings buffer is nearly or completely full and you need to raise it to maintain its full performance benefit, or ignore the recommendation if RAM size became a limit/performance decrease at the other end.

Since we still didn't find an app/apps which can be made responsible for such a large strings buffer usage in a single (and no other web application) Nextcloud (more than 16 MiB, which was found to be very rare), you may post your apps list, like so:

<details>
<summary>My apps list</summary>

<output of "occ app:list">

</details>

When I find time, I'll try to find/understand which/why apps can use that much of this buffer, and if it is really due to temporary large PHP files (my only idea, so far), if/how related strings can be evicted from the buffer without clearing the whole OPcache.

slawa-dev commented 2 years ago

@MichaIng One instance of NextCloud (nginx + PHP FPM 7.4 + MariaDB), an instance of Matomo (https://github.com/matomo-org/matomo) and a few cron jobs are running. All barely used, server is mostly idle. It has been running like that for a long time. The OPcache warning appeared just recently.

MichaIng commented 2 years ago

... As I said:

Guys, there is no need to keep posting when you see this warning.

And running another web application like Matomo of course is a pretty good reason for additional buffer usage, so it is really expected then and the recommendation does exactly was it is supposed to, so please do not post here in such case. The only case where additional information would be interesting is when a single Nextcloud is the only web application on your PHP instance and you additionally append your apps list.