search

nextcloud / all-in-one

📦 The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance.
https://hub.docker.com/r/nextcloud/all-in-one
GNU Affero General Public License v3.0
5.66k stars 658 forks source link

The PHP OPcache module is not properly configured #5559

Open mitexleo opened 2 weeks ago

mitexleo commented 2 weeks ago

Steps to reproduce

  1. Upgrade Nextcloud Manually to the latest version
  2. Check Overview Page

Expected behavior

Everything should work out of the box.

Actual behavior

Image

Other information

Host OS Ubuntu 24 LTS

Output of sudo docker info

Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server: Containers: 22 Running: 21 Paused: 0 Stopped: 1 Images: 22 Server Version: 27.3.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-48-generic Operating System: Ubuntu 24.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.57GiB Name: mastoleo ID: 667551f2-25b0-4656-8585-f9aeba82558f Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled

Docker run command or docker-compose file that you used

Other valuable info

szaimen commented 2 weeks ago

Cc @MichaIng

MichaIng commented 2 weeks ago

@szaimen there is no OPcache usage monitor built into the container image, is it? Would be actually a great idea for an app, visualising opcache_get_status output, or having this in Monitoring app.

Else @mitexleo are you open to access container console and put a script with the output of above function into your container, or https://github.com/amnuts/opcache-gui alternatively, to check which scripts use that much OPcache?

szaimen commented 2 weeks ago

there is no OPcache usage monitor built into the container image, is it?

There isnt unfortunately.

Would be actually a great idea for an app, visualising opcache_get_status output, or having this in Monitoring app.

I agree. This could be a feature request for https://github.com/nextcloud/serverinfo. WDYT? :)

szaimen commented 2 weeks ago

Would be actually a great idea for an app, visualising opcache_get_status output, or having this in Monitoring app.

I agree. This could be a feature request for https://github.com/nextcloud/serverinfo. WDYT? :)

@MichaIng I had a fast look and it looks like this is actually already implemented and visible when you open https://your-nc-domain.com/ocs/v2.php/apps/serverinfo/api/v1/info?format=json

szaimen commented 2 weeks ago

On my test server I get this output:

          "opcache_revalidate_freq": 60,
          "opcache": {
            "opcache_enabled": true,
            "cache_full": false,
            "restart_pending": false,
            "restart_in_progress": false,
            "memory_usage": {
              "used_memory": 138093632,
              "free_memory": 130341824,
              "wasted_memory": 0,
              "current_wasted_percentage": 0
            },
            "interned_strings_usage": {
              "buffer_size": 67108864,
              "used_memory": 23571008,
              "free_memory": 43537856,
              "number_of_strings": 105021
            },
            "opcache_statistics": {
              "num_cached_scripts": 2660,
              "num_cached_keys": 5083,
              "max_cached_keys": 16229,
              "hits": 16540251,
              "start_time": 1731125044,
              "last_restart_time": 0,
              "oom_restarts": 0,
              "hash_restarts": 0,
              "manual_restarts": 0,
              "misses": 2711,
              "blacklist_misses": 0,
              "blacklist_miss_ratio": 0,
              "opcache_hit_rate": 99.9836123663949
            },
            "jit": {
              "enabled": true,
              "on": true,
              "kind": 5,
              "opt_level": 5,
              "opt_flags": 6,
              "buffer_size": 8388592,
              "buffer_free": 7332352
            }
          },
MichaIng commented 2 weeks ago

Ah nice. I just checked, sadly there is no way yet to show a list of cached keys/scripts, as function argument false is hardcoded: https://github.com/nextcloud/serverinfo/blob/master/lib/PhpStatistics.php#L56 I do not expect the Nextcloud checks to be wrong, hence it would be interesting to see which scripts/apps make most use of the cache. For that, one would need to call opcache_get_status(true) or opcache_get_status(). Maybe I am able to implement a new argument, based on how it is done for skipping system/app update infos: https://github.com/nextcloud/serverinfo/blob/master/lib/Controller/ApiController.php#L101 It would however make the JSON veeery long, with several thousand script paths 😄.

szaimen commented 2 weeks ago

One could temporarily hack the phpstatistics class on their server...

szaimen commented 2 weeks ago

@mitexleo would you mind opening https://your-nc-domain.com/ocs/v2.php/apps/serverinfo/api/v1/info?format=json as admin in your browser and posting the requested stats about opcache here?

mitexleo commented 2 weeks ago

@mitexleo would you mind opening https://your-nc-domain.com/ocs/v2.php/apps/serverinfo/api/v1/info?format=json as admin in your browser and posting the requested stats about opcache here?

This is what I get:

"opcache_revalidate_freq": 60, "opcache": { "opcache_enabled": true, "cache_full": false, "restart_pending": false, "restart_in_progress": false, "memory_usage": { "used_memory": 222317712, "free_memory": 7507832, "wasted_memory": 38609912, "current_wasted_percentage": 14.3833130598068 }, "interned_strings_usage": { "buffer_size": 67108864, "used_memory": 28571384, "free_memory": 38537480, "number_of_strings": 178384 }, "opcache_statistics": { "num_cached_scripts": 4625, "num_cached_keys": 8902, "max_cached_keys": 16229, "hits": 38995048, "start_time": 1731039835, "last_restart_time": 0, "oom_restarts": 0, "hash_restarts": 0, "manual_restarts": 0, "misses": 7265, "blacklist_misses": 0, "blacklist_miss_ratio": 0, "opcache_hit_rate": 99.9813728996021 },
szaimen commented 1 week ago

@MichaIng does the above help?

MichaIng commented 1 week ago

As said, the stats/conclusion is correct. Interesting would be the actual scripts which use up that much space: https://github.com/nextcloud/all-in-one/issues/5559#issuecomment-2466502800

So for now it'd require an additional script within webroot with opcache_get_status(true), or https://github.com/amnuts/opcache-gui.

szaimen commented 1 week ago

Hi @mitexleo can you please run sudo docker exec nextcloud-aio-nextcloud sed -i 's|opcache_get_status(false)|opcache_get_status(true)|' apps/serverinfo/lib/PhpStatistics.php and then wait 60s and then open the link again and post the new output here? Thanks a lot! :)

mitexleo commented 1 week ago

Hi @mitexleo can you please run sudo docker exec nextcloud-aio-nextcloud sed -i 's|opcache_get_status(false)|opcache_get_status(true)|' apps/serverinfo/lib/PhpStatistics.php and then wait 60s and then open the link again and post the new output here? Thanks a lot! :)

Got too long logs:

{"ocs":{"meta":{"status":"ok","statuscode":200,"message":"OK"},"data":{"nextcloud":{"system":{"version":"30.0.2.2","theme":"none","enable_avatars":"yes","enable_previews":"yes","memcache.local":"\\OC\\Memcache\\APCu","memcache.distributed":"\\OC\\Memcache\\Redis","filelocking.enabled":"yes","memcache.locking":"\\OC\\Memcache\\Redis","debug":"no","freespace":null,"cpuload":[5.1591796875,15.9541015625,13.97412109375],"mem_total":7937024,"mem_free":1690624,"swap_total":4193280,"swap_free":23552},"storage":{"num_users":4,"num_files":134270,"num_storages":9,"num_storages_local":1,"num_storages_home":2,"num_storages_other":6},"shares":{"num_shares":416,"num_shares_user":1,"num_shares_groups":0,"num_shares_link":17,"num_shares_mail":2,"num_shares_room":175,"num_shares_link_no_password":17,"num_fed_shares_sent":0,"num_fed_shares_received":0,"permissions_10_19":175,"permissions_3_17":17,"permissions_11_19":219,"permissions_4_20":2,"permissions_0_17":1,"permissions_11_0":2}},"server":{"webserver":"Apache/2.4.62 (Unix)","php":{"version":"8.2.25","memory_limit":1073741824,"max_execution_time":3600,"upload_max_filesize":10737418240,"opcache_revalidate_freq":60,"opcache":{"opcache_enabled":true,"cache_full":false,"restart_pending":false,"restart_in_progress":false,"memory_usage":{"used_memory":224051208,"free_memory":4153320,"wasted_memory":40230928,"current_wasted_percentage":14.987188577652},"interned_strings_usage":{"buffer_size":67108864,"used_memory":28620552,"free_memory":38488312,"number_of_strings":179018},"opcache_statistics":{"num_cached_scripts":4672,"num_cached_keys":8994,"max_cached_keys":16229,"hits":52178560,"start_time":1731039835,"last_restart_time":0,"oom_restarts":0,"hash_restarts":0,"manual_restarts":0,"misses":7398,"blacklist_misses":0,"blacklist_miss_ratio":0,"opcache_hit_rate":99.9858237727475},"scripts":{"/var/www/html/apps/files_versions/lib/Listener/VersionAuthorListener.php":
full_path":"/var/www/html/apps/files_versions/lib/Listener/VersionAuthorListener.php","hits":238,"memory_consumption":5088,"last_used":"Wed Nov 13 18:26:59 2024","last_used_timestamp":1731522419,"timestamp":1731040359},"/var/www/html/apps/dav/lib/CalDAV/Activity/Filter/Calendar.php":{"full_path":"/var/www/html/apps/dav/lib/CalDAV/Activity/Filter/Calendar.php","hits":30,"memory_consumption":6264,"last_used":"Thu Nov 14 00:31:20 2024","last_used_timestamp":1731544280,"timestamp":1731040360},"/var/www/html/apps/text/lib/Db/Session.php":{"full_path":"/var/www/html/apps/text/lib/Db/Session.php","hits":5,"memory_consumption":5800,"last_used":"Sun Nov 10 17:38:45 2024","last_used_timestamp":1731260325,"timestamp":1731040361},"/var/www/html/custom_apps/richdocuments/lib/Controller/DirectViewController.php":{"full_path":"/var/www/html/custom_apps/richdocuments/lib/Controller/DirectViewController.php","hits":9488,"memory_consumption":18888,"last_used":"Thu Nov 14 00:36:23 2024","last_used_timestamp":1731544583,"timestamp":1729815382},"/var/www/html/custom_apps/polls/lib/Controller/BaseController.php":{"full_path":"/var/www/html/custom_apps/polls/lib/Controller/BaseController.php","hits":1364,"memory_consumption":8576,"last_used":"Thu Nov 14 00:33:59 2024","last_used_timestamp":1731544439,"timestamp":1727378807},"/var/www/html/custom_apps/metadata/appinfo/routes.php":{"full_path":"/var/www/html/custom_apps/metadata/appinfo/routes.php","hits":1345,"memory_consumption":1088,"last_used":"Thu Nov 14 00:33:59 2024","last_used_timestamp":1731544439,"timestamp":1728146388},"/var/www/html/3rdparty/aws/aws-sdk-php/src/Result.php":{"full_path":"/var/www/html/3rdparty/aws/aws-sdk-php/src/Result.php","hits":31135,"memory_consumption":5320,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040358},"/var/www/html/lib/public/Search/Provider.php":{"full_path":"/var/www/html/lib/public/Search/Provider.php","hits":27,"memory_consumption":5616,"last_used":"Thu Nov 14 00:31:50 2024","last_used_timestamp":1731544310,"timestamp":1731040362},"/var/www/html/custom_apps/collectives/vendor/teamtnt/tntsearch/helper/helpers.php":{"full_path":"/var/www/html/custom_apps/collectives/vendor/teamtnt/tntsearch/helper/helpers.php","hits":31635,"memory_consumption":3656,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1730992646},"/var/www/html/lib/public/Preview/IProviderV2.php":{"full_path":"/var/www/html/lib/public/Preview/IProviderV2.php","hits":31423,"memory_consumption":3520,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040362},"/var/www/html/apps/user_status/lib/Connector/UserStatus.php":{"full_path":"/var/www/html/apps/user_status/lib/Connector/UserStatus.php","hits":170,"memory_consumption":7376,"last_used":"Thu Nov 14 00:31:52 2024","last_used_timestamp":1731544312,"timestamp":1731040361},"/var/www/html/apps/webhook_listeners/lib/Db/WebhookListenerMapper.php":{"full_path":"/var/www/html/apps/webhook_listeners/lib/Db/WebhookListenerMapper.php","hits":31134,"memory_consumption":26912,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040359},"/var/www/html/lib/private/App/AppStore/Version/VersionParser.php":{"full_path":"/var/www/html/lib/private/App/AppStore/Version/VersionParser.php","hits":15,"memory_consumption":5944,"last_used":"Thu Nov 14 00:31:54 2024","last_used_timestamp":1731544314,"timestamp":1731040362},"/var/www/html/custom_apps/collectives/lib/Listeners/CollectivesReferenceListener.php":{"full_path":"/var/www/html/custom_apps/collectives/lib/Listeners/CollectivesReferenceListener.php","hits":39,"memory_consumption":2656,"last_used":"Thu Nov 14 00:31:43 2024","last_used_timestamp":1731544303,"timestamp":1730992646},"/var/www/html/apps/circles/lib/Tools/Traits/TNCWellKnown.php":{"full_path":"/var/www/html/apps/circles/lib/Tools/Traits/TNCWellKnown.php","hits":31135,"memory_consumption":8960,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040361},"/var/www/html/custom_apps/tables/vendor/ezyang/htmlpurifier/library/HTMLPurifier/AttrDef/HTML/ID.php":{"full_path":"/var/www/html/custom_apps/tables/vendor/ezyang/htmlpurifier/library/HTMLPurifier/AttrDef/HTML/ID.php","hits":759,"memory_consumption":7072,"last_used":"Thu Nov 14 00:31:50 2024","last_used_timestamp":1731544310,"timestamp":1728930978},"/var/www/html/apps/files/lib/Activity/Helper.php":{"full_path":"/var/www/html/apps/files/lib/Activity/Helper.php","hits":30,"memory_consumption":7296,"last_used":"Thu Nov 14 00:31:20 
2024","last_used_timestamp":1731544280,"timestamp":1731040360},"/var/www/html/custom_apps/user_migration/appinfo/routes.php":{"full_path":"/var/www/html/custom_apps/user_migration/appinfo/routes.php","hits":1346,"memory_consumption":2224,"last_used":"Thu Nov 14 00:33:59 2024","last_used_timestamp":1731544439,"timestamp":1728147200},"/var/www/html/apps/files_sharing/composer/composer/autoload_static.php":{"full_path":"/var/www/html/apps/files_sharing/composer/composer/autoload_static.php","hits":31422,"memory_consumption":7552,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040359},"/var/www/html/apps/dav/lib/CalDAV/Activity/Setting/Calendar.php":{"full_path":"/var/www/html/apps/dav/lib/CalDAV/Activity/Setting/Calendar.php","hits":280,"memory_consumption":4872,"last_used":"Thu Nov 14 00:31:20 2024","last_used_timestamp":1731544280,"timestamp":1731040360},"/var/www/html/custom_apps/collectives/lib/Db/CollectiveUserSettingsMapper.php":{"full_path":"/var/www/html/custom_apps/collectives/lib/Db/CollectiveUserSettingsMapper.php","hits":31204,"memory_consumption":9096,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1730992646},"/var/www/html/custom_apps/deck/lib/Db/RelationalEntity.php":{"full_path":"/var/www/html/custom_apps/deck/lib/Db/RelationalEntity.php","hits":5,"memory_consumption":13000,"last_used":"Thu Nov 14 00:36:14 2024","last_used_timestamp":1731544574,"timestamp":1731544428},"/var/www/html/custom_apps/news/lib/Fetcher/FaviconDataAccess.php":{"full_path":"/var/www/html/custom_apps/news/lib/Fetcher/FaviconDataAccess.php","hits":761,"memory_consumption":5224,"last_used":"Thu Nov 14 00:31:50 2024","last_used_timestamp":1731544310,"timestamp":1729002702},"/var/www/html/apps/files_external/lib/Service/GlobalStoragesService.php":{"full_path":"/var/www/html/apps/files_external/lib/Service/GlobalStoragesService.php","hits":31134,"memory_consumption":12792,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040361},"/var/www/html/custom_apps/collectives/lib/Listeners/BeforeTemplateRenderedListener.php":{"full_path":"/var/www/html/custom_apps/collectives/lib/Listeners/BeforeTemplateRenderedListener.php","hits":187,"memory_consumption":6528,"last_used":"Thu Nov 14 00:31:49 2024","last_used_timestamp":1731544309,"timestamp":1730992646},"/var/www/html/custom_apps/tasks/lib/Controller/PageController.php":{"full_path":"/var/www/html/custom_apps/tasks/lib/Controller/PageController.php","hits":1363,"memory_consumption":4160,"last_used":"Thu Nov 14 00:33:59 2024","last_used_timestamp":1731544439,"timestamp":1726745963},"/var/www/html/lib/private/KnownUser/KnownUserMapper.php":{"full_path":"/var/www/html/lib/private/KnownUser/KnownUserMapper.php","hits":31130,"memory_consumption":8088,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040362},"/var/www/html/lib/private/Preview/Movie.php":{"full_path":"/var/www/html/lib/private/Preview/Movie.php","hits":1,"memory_consumption":13640,"last_used":"Mon Nov 11 14:35:25 2024","last_used_timestamp":1731335725,"timestamp":1731040362},"/var/www/html/apps/dav/lib/CalDAV/IRestorable.php":{"full_path":"/var/www/html/apps/dav/lib/CalDAV/IRestorable.php","hits":118,"memory_consumption":1848,"last_used":"Thu Nov 14 00:31:29 2024","last_used_timestamp":1731544289,"timestamp":1731040360},"/var/www/html/config/proxy.config.php":{"full_path":"/var/www/html/config/proxy.config.php","hits":31639,"memory_consumption":2072,"last_used":"Thu Nov 14 00:36:49 
2024","last_used_timestamp":1731544609,"timestamp":1726745853},"/var/www/html/apps/files_external/lib/Service/UserStoragesService.php":{"full_path":"/var/www/html/apps/files_external/lib/Service/UserStoragesService.php","hits":31132,"memory_consumption":11416,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040361},"/var/www/html/custom_apps/recognize/lib/Settings/AdminSettings.php":{"full_path":"/var/www/html/custom_apps/recognize/lib/Settings/AdminSettings.php","hits":12,"memory_consumption":4840,"last_used":"Tue Nov 12 12:29:14 2024","last_used_timestamp":1731414554,"timestamp":1728930617},"/var/www/html/custom_apps/groupfolders/lib/Controller/DelegationController.php":{"full_path":"/var/www/html/custom_apps/groupfolders/lib/Controller/DelegationController.php","hits":1350,"memory_consumption":9744,"last_used":"Thu Nov 14 00:33:59 2024","last_used_timestamp":1731544439,"timestamp":1728930601},"/var/www/html/custom_apps/news/lib/Listeners/AddMissingIndicesListener.php":{"full_path":"/var/www/html/custom_apps/news/lib/Listeners/AddMissingIndicesListener.php","hits":7,"memory_consumption":2392,"last_used":"Tue Nov 12 12:22:41 2024","last_used_timestamp":1731414161,"timestamp":1729002702},"/var/www/html/3rdparty/sabre/http/lib/RequestInterface.php":{"full_path":"/var/www/html/3rdparty/sabre/http/lib/RequestInterface.php","hits":2290,"memory_consumption":9328,"last_used":"Thu Nov 14 00:34:18 2024","last_used_timestamp":1731544458,"timestamp":1731040357},"/var/www/html/custom_apps/tables/vendor/ezyang/htmlpurifier/library/HTMLPurifier/AttrDef/Clone.php":{"full_path":"/var/www/html/custom_apps/tables/vendor/ezyang/htmlpurifier/library/HTMLPurifier/AttrDef/Clone.php","hits":761,"memory_consumption":3928,"last_used":"Thu Nov 14 00:31:50 2024","last_used_timestamp":1731544310,"timestamp":1728930978},"/var/www/html/3rdparty/symfony/css-selector/Parser/Shortcut/HashParser.php":{"full_path":"/var/www/html/3rdparty/symfony/css-selector/Parser/Shortcut/HashParser.php","hits":2,"memory_consumption":3480,"last_used":"Mon Nov 11 15:15:12 2024","last_used_timestamp":1731338112,"timestamp":1731040357},"/var/www/html/apps/files_external/lib/Config/IConfigHandler.php":{"full_path":"/var/www/html/apps/files_external/lib/Config/IConfigHandler.php","hits":31135,"memory_consumption":1968,"last_used":"Thu Nov 14 00:36:49 2024","last_used_timestamp":1731544609,"timestamp":1731040361},"/var/www/html/lib/public/Files/SimpleFS/InMemoryFile.php":{"full_path":"/var/www/html/lib/public/Files/SimpleFS/InMemoryFile.php","hits":73,"memory_consumption":9352,"last_used":"Thu Nov 14 00:31:33 2024","last_used_timestamp":1731544293,"timestamp":1731040362},"/var/www/html/lib/private/Preview/Image.php":{"full_path":"/var/www/html/lib/private/Preview/Image.php","hits":392,"memory_consumption":3832,"last_used":"Thu Nov 14 00:31:25 2024","last_used_timestamp":1731544285,"timestamp":1731040362},"/var/www/html/lib/public/Translation/LanguageTuple.php":{"full_path":"/var/www/html/lib/public/Translation/LanguageTuple.php","hits":38,"memory_consumption":4616,"last_used":"Thu Nov 14 00:31:43 2024","last_used_timestamp":1731544303,"timestamp":1731040362},"/var/www/html/3rdparty/sabre/dav/lib/DAVACL/IPrincipal.php":{"full_path":"/var/www/html/3rdparty/sabre/dav/lib/DAVACL/IPrincipal.php","hits":151,"memory_consumption":5064,"last_used":"Thu Nov 14 00:26:42 2024","last_used_timestamp":1731544002,"timestamp":1731040357},"/var/www/html/lib/private/App/AppStore/Fetcher/Fetcher.php":

Is there any other way to share the logs? Maybe I could give you access to the server?

szaimen commented 1 week ago

Maybe you can use something like https://pastebin.com/

Or put your content into a file and upload the file here?

mitexleo commented 1 week ago

Maybe you can use something like https://pastebin.com/

Or put your content into a file and upload the file here?

There you go: https://cloud.buddyverse.one/s/TTFGenwsrPB4rgD

szaimen commented 1 week ago

@MichaIng does the above help now? :)

pagenstedt commented 1 week ago

Steps to reproduce

  1. Upgrade Nextcloud Manually to the latest version
  2. Check Overview Page

I did have the same issue. A restart of the docker container fixes the opcache warning.

mitexleo commented 1 week ago

Steps to reproduce

  1. Upgrade Nextcloud Manually to the latest version
  2. Check Overview Page

I did have the same issue. A restart of the docker container fixes the opcache warning.

Confirmed. This does fix the issue!

MichaIng commented 1 week ago

I extracted the scripts list with memory consumption as array: info.json

Is someone good at querying it for summed up memory consumption per directory? On first sight, 3rdparty/aws/aws-sdk-php seems to consume quite some space, which I do not have cached in any of my instances. Which app/feature does make use of AWS access? The mail app is also larger, but I just installed it, and first access caches even more of its sripts than in above list, includig all large ones, but still no significant change overall in my case. Other large apps like spreed (Nextcloud Talk) are installed on my instances as well, but overall used memory is still only 95 MiB on the largest one.

Restarting the container is likely only a temporary solution.

mitexleo commented 6 days ago

I extracted the scripts list with memory consumption as array: info.json

Is someone good at querying it for summed up memory consumption per directory? On first sight, 3rdparty/aws/aws-sdk-php seems to consume quite some space, which I do not have cached in any of my instances. Which app/feature does make use of AWS access? The mail app is also larger, but I just installed it, and first access caches even more of its sripts than in above list, includig all large ones, but still no significant change overall in my case. Other large apps like spreed (Nextcloud Talk) are installed on my instances as well, but overall used memory is still only 95 MiB on the largest one.

Restarting the container is likely only a temporary solution.

I'm using S3 as primary storage backend. Could it be an issue?

MichaIng commented 6 days ago

The sum of all cached script sizes is 72 MiB + 64 MiB interned strings buffer + 8 MiB JIT + 39 MiB wasted memory = 183 MiB. We know JIT somehow raises the overall OPcache memory usage more than the 1 MiB it occupies from the 8 MiB max JIT buffer size, and also more than the 8 MiB. Here the diff is 73 MiB, quite close to the 72 MiB cached script size.

Let me check the summed script size of my instances ...

... one has 46 MiB cached scripts + 16 MiB interned strings buffer + 2 MiB JIT (1 MiB used) = 64 MiB, showing 96 MiB overall used OPcache buffer size => 32 MiB diff.

Another one is ARM, hence no JIT, with 28 MiB cached scripts + 8 MiB interned strings buffer = 36 MiB, showing 39 MiB overall used OPcache buffer size => only 3 MiB diff.

Another PHP server which has no Nextcloud but JIT, running Wordpress + Matomo, has 64 MiB cached scripts ... oh 128 MiB fully used interned strings buffer + 2 MiB JIT (1 MiB used) + 12 MiB wasted memory = 186 MiB, showing 244 MiB overall used => 58 MiB diff.

All cases with JIT show the non-transparently raised overall OPcache usage, while the defined JIT buffer size is always occupied by only ~1 MiB. It seems to raise by moreless the same size, the opcode of the scripts have, i.e. scripts seem to occupy about the doubled size in overall OPcache.

So in your case @mitexleo, you have more cached scripts than my instances, the S3 backend causing part of it, but in sum 5 MiB only, and overall 72 MiB vs 46 MiB is not a huge difference. Then you have 39 MiB wasted memory, which is relatively high, but I assume PHP to do garbage collection if needed. Then the interned strings buffer is chosen large, which occupies OPcache memory always in full size right from the start, but that is chosen as it is eaten up on some instances without limits (see also my Wordpress instance), hence we disabled the warning about it Nextcloud prints, if its size is 25% of overall OPcache size already. This is why AIO chose 64 MiB interned strings buffer size, being 25% of 256 MiB overall OPcache size. And the largest part seems to be the roughly doubled usage of each cached script, due to enabled JIT.

@szaimen not sure whether there are more reports about this. Of course OPcache sizes could be doubled again, but maybe it makes sense to have them either as variable for container setup/run, or even have a dedicated PHP drop-in/module config outside the container in a volume, to be edited by admins?

Another idea I have, is considering the "wasted" OPcache memory in Nextcloud setup checks as free space, expecting that PHP would clean it up if needed. But I don't know how it is really handled internally. Apart of the non-transparent JIT buffer usage, something check back with PHP/OPcache devs.

szaimen commented 6 days ago

@pagenstedt are you also using s3 primary storage?

szaimen commented 6 days ago

@MichaIng would it help to disable jit again? Even though it might come with some (hopefully small?) performance implications?

MichaIng commented 6 days ago

I guess that would help, I could do some test to verify this on my instances, i.e. whether this invisibly added OPcache usage really goes away then, and re-appears once re-enabling JIT. But would be a shame to run an x86 PHP 8.x instance without its biggest advantage enabled, isn't it? I mean Nextcloud warns about full OPcache buffer, to help admins optimise their Nextcloud performance. Ridiculous to mute these warnings by disabling the probably biggest performance boost that PHP offers nowadays 😄. Then better either raise OPcache size again, or implement a way into Nextcloud to disable this check for appliances (which naturally are not as flexible but must match a wide range of use cases). And taking into account "wasted" memory seems to be quite reasonable as well. I'll dig a little into this part.

MichaIng commented 6 days ago

https://sergey-lysenko.com/opcache-wasted-memory/

So this grows when scripts are changed. But resetting the cache opcache_invalidate(), clears wasted memory as well. So I guess this is mostly relevant when upgrading Nextcloud or apps via CLI. In this case scripts change, but the CLI PHP instance cannot clear the web instance cache. That one just checks the files as part of regular revalidate, and in case of a change, seems to leave behind the old opcode as wasted memory.

https://www.php.net/manual/en/opcache.configuration.php#ini.opcache.max-wasted-percentage

So OPcache schedules a restart (I guess an OPcache reset?), if wasted memory raises above 5% (by default), and free memory is insufficient. Theoretically, Nextcloud could check this setting, and assume wasted memory above this size as free memory, for its check. Or, to simplify, just assume all wasted memory as free memory. Not sure how much complexity we want in the check 😄.

szaimen commented 6 days ago

Thanks a lot for investigating!

Or, to simplify, just assume all wasted memory as free memory.

This sounds like a good idea to me! :)

MichaIng commented 5 days ago

I opened an issue at server repo: https://github.com/nextcloud/server/issues/49438

szaimen commented 5 days ago

Thanks a lot! :)