opcache fills and becomes non-functional

[dotdash]

• Deployer version: 7.4
• Deployment OS: Ubuntu 20.04

This is potentially a documentation issue, as https://deployer.org/docs/7.x/avoid-php-fpm-reloading and https://deployer.org/docs/7.x/contrib/php-fpm both say that a reload is not needed the server has been provisioned by Deployer. But this only holds true until the cache becomes full.

The setup performed by the provisioning recipe does indeed prevent the problems that might arise from simply switching the symlink where the cache might see the same path again and thus become disfunctional right away, but as there is no mechanism that purges the cache, it still becomes useless once it is full.

The problem is, that the cache never drops its contents until there is are enough invalidated entries to consume more than opcache.max_wasted_percentage percent of the cache. But invalidation only happens when files are accessed. As the files from previous deployments do not get accesssed anymore, they don't get invalidated and therefore the cache never resets itself. And after enough deployments, it becomes completely useless.

With the default setup, after deploying the below about 10 times, and accessing index.php each time, the opcache becomes full, and further deployments run fully uncached.

var_dump of opcache_get_status() after ~12 deployments and a bunch of reloads each time:

array(8) {
  ["opcache_enabled"]=>
  bool(true)
  ["cache_full"]=>
  bool(true)
  ["restart_pending"]=>
  bool(false)
  ["restart_in_progress"]=>
  bool(false)
  ["memory_usage"]=>
  array(4) {
    ["used_memory"]=>
    int(14779624)
    ["free_memory"]=>
    int(119438104)
    ["wasted_memory"]=>
    int(0)
    ["current_wasted_percentage"]=>
    float(0)
  }
  ["interned_strings_usage"]=> // not relevant
  ["opcache_statistics"]=>
  array(13) {
    ["num_cached_scripts"]=>
    int(8123)
    ["num_cached_keys"]=>
    int(16229)
    ["max_cached_keys"]=>
    int(16229)
    ["hits"]=>
    int(50205)
    ["start_time"]=>
    int(1718716995)
    ["last_restart_time"]=>
    int(1718719950)
    ["oom_restarts"]=>
    int(0)
    ["hash_restarts"]=>
    int(0)
    ["manual_restarts"]=>
    int(1)
    ["misses"]=>
    int(93018)
    ["blacklist_misses"]=>
    int(0)
    ["blacklist_miss_ratio"]=>
    float(0)
    ["opcache_hit_rate"]=>
    float(35.05372740411805)
  }
  ["jit"]=> // not relevant
}

as you can see, the cache is full, and none of the cache entries are invalidated (wasted_memory is 0), so no restart gets planned.

I'd argue that the documentation should at least mention this and possibly refer to e.g. cachetool to rectify the situation. I'd also like some clarification in which circumstances a reload (as opposed to a restart) would lead to broken/lost requests, AFAIK the reload works as a graceful restart, finishing currently running requests and queuing outstanding requests until the restart is done.

index.php:

<?php
header('Content-type: text/plain');
var_dump(opcache_get_status(false));

$mustwait = false;
for ($i = 1; $i <= 1000; $i++) {
        if (!file_exists("../inc/foo$i.php")) {
                file_put_contents("../inc/foo$i.php", '');
                $mustwait = true;
        }
}
if ($mustwait) {
        sleep(5); // Wait so opcache treats the file as cachable
}

for ($i = 1; $i <= 1000; $i++) {
        @include "../inc/foo$i.php";
}

deploy.php:

<?php
namespace Deployer;

require 'recipe/composer.php';
require 'recipe/provision.php';

// Config

set('repository', '/root/repo');

add('shared_files', []);
add('shared_dirs', []);
add('writable_dirs', []);

// Hosts

host('172.17.0.2')
  ->set('remote_user', 'root')
  ->set('deploy_path', '~/derp');

// Hooks

task('provision:firewall', function () {}); // Running in docker, the fw setup failed

after('deploy:failed', 'deploy:unlock');

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.