Feature request: Transparent SHM reuse

[dstogov]

OPcache should reuse shared memory previously allocated by invalidated scripts so expensive cache reset wouldn't be necessary anymore.

[TerryE]

I've been thinking about the issues around this one for a few weeks now. (This is only one of the resource limits that can cause SHM exhaustion or degradation -- albeit the largest single one. The others are the static sizing of the interned pool, and the two hash tables in the SHM). The key issue here isn't the reuse implementation itself as there are many robust patterns that could be a basis for this; it's that the running scripts use absolute address referencing into scripts which must be preserved for the life of all requests using that compiled script, even after invalidation. Failure to do this will lead to executor crashes. So some form of per compiled script read lock is required so that the storage layer could determine when the invalidated storage can safely be reused. A per request/script FLOCK implementation would be a real performance hit.

Unless there are any other priority takers, I'll write up the issues and suggested approach in whitepaper format and link back here. Depending of feedback and general comfort level I can then move onto implementation.

[TerryE]

One clarification on this. For the above reasons, I don't think that implementing a robust reuse model is something that can be considered as a quick 5.5.x change. It's side-effects are just too wide ranging. However, doing this properly in 5.6 timescales is entirely feasible. What might be sensible is to get a demonstrator 5.5 fork done to evaluate the soundness and performance of the approach and then integrate this into the mainline 5.6 development.

[dstogov]

Note, that the implementation would probably require more locks and could be slower.

On Sun, Aug 4, 2013 at 9:26 PM, Terry Ellison notifications@github.comwrote:

One clarification on this. For the above reasons, I don't think that implementing a robust reuse model is something that can be considered as a quick 5.5.x change. It's side-effects are just too wide ranging. However, doing this properly in 5.6 timescales is entirely feasible. What might be sensible is to get a demonstrator 5.5 fork done to evaluate the soundness and performance of the approach and then integrate this into the mainline 5.6 development.

— Reply to this email directly or view it on GitHubhttps://github.com/zendtech/ZendOptimizerPlus/issues/118#issuecomment-22075261 .

[jfschedler]

What is the status of SHM re-use for php 5.6? Is there a white paper on this? We really need this feature to keep our continuous deployment architecture running.

[rlerdorf]

@jfschedler it is quite easy to do continuous deployment with Opcache. Granted, if your codebase is large you need a big chunk, but I have 200+ developers on a huge codebase pushing to production 30-40 times per day without any issues. The shm is cleared nightly on the logrotate restart, but that is the only time I clear it. You can see a full description of the deploy procedure here:

http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

[dstogov]

sorry, all our forces are on phpng development now. see https://wiki.php.net/phpng I'm not sure, if anyone able to implement SHM re-use and fit into PHP-5.6 release cycle.

Thanks. Dmitry.

On Mon, May 19, 2014 at 7:33 PM, Jon Schedler notifications@github.comwrote:

What is the status of SHM re-use for php 5.6? Is there a white paper on this? We really need this feature to keep our continuous deployment architecture running.

— Reply to this email directly or view it on GitHubhttps://github.com/zendtech/ZendOptimizerPlus/issues/118#issuecomment-43519321 .

[sparc]

it seems the same as #187 , at least would be great that calls to opcache_invalidate could free the invalidated script used memory