Zend Opcache Failure - Unable to reattach to base address / Attempt to access invalid address.

[laurinkeithdavis]

I'm not sure what is going on, but we just upgraded to PHP 5.5.9 from PHP 5.5.8 and we are now experiencing this error in our Event Log (Windows.)

Event ID 487 from source Zend Opcache Unable to reattach to base address Attempt to access invalid address.

I see there is a directive that is supposed to "fix" this problem opcache.mmap_base, however, we did not have this problem prior to today. I'm not certain that 5.5.9 is the problem, but I am rolling back, just to test for a few days. The only other thing that changed, was we installed a number of Windows Updates.

EDIT: Ok, so it did not begin yesterday. I got this error 4 times in October, then starting 9 days ago, it occurred about once a day, however, starting yesterday, it is occurring 10 times a day. Rolling back to 5.5.8 did not resolve the increased frequency, it appears, as I'm the only one on the server this morning and the second page I opened crashed.

[szarkos]

Looks related to these two issues: https://bugs.php.net/bug.php?id=64926 https://github.com/zendtech/ZendOptimizerPlus/issues/109

Can you test opcache.mmap_base to see if that helps? The bug report above tried the following: opcache.mmap_base = 0x20000000

I think this needs to be fixed by 5.6 timeframe.

[laurinkeithdavis]

I actually was testing that, but with the wrong value (and it failed once since then.) I've changed it to the value you indicate. Now, we wait and see...

[laurinkeithdavis]

Well, it has worked fine until just now. To perform an update, we did iisreset and then because we had an issue had to restart the server.

[laurinkeithdavis]

It's almost unusable right now. I might just have to disable the Opcache until a solution is found. Right now, I'm trying alternate values per the Zend page indicated.

FYI, the link on PHP.net is bad. Here is the actual link:

http://kb.zend.com/zend-enabler-error-could-not-locate-a-free-php-process/#.UwYNwViA1aQ

[laurinkeithdavis]

I've tried every value available and of right now, we are disabling Opcache completely - it's unusable.

[laurinkeithdavis]

FYI, with this value 0×50000000, it fails constantly - the others work most of the time, but still fails to frequently to be usable.

[szarkos]

Just curious, can you share more details about your IIS configuration; how many sites/app pools or fastcgi processes are typically running, etc.? Have you configured your FastCGI settings similar to this or are they significantly different: http://www.iis.net/learn/application-frameworks/install-and-configure-php-applications-on-iis/using-fastcgi-to-host-php-applications-on-iis#PHP_Recycling_Behavior

It seems this issue can occur during startup of a FastCGI process. On process startup if it fails to obtain the base address it will die. Possibly reducing the frequency of process recycling might help here.

From the comment above sounded like it started to fail hard after you updated and restarted IIS? One thing to try, whenever I restart IIS I usually delete the Wincache and Opcache files in the temp directory as well (I used to do this before Opcache anyway). So for example:

iisreset /stop del c:\windows\temp\ZendOPcache* iisreset /start

c:\windows\temp is the default, but that may vary depending on your configuration.

[laurinkeithdavis]

3 application pools are normally in use. My FastCGI is configured exactly per the instructions in that URL (used those to actually configure the settings.)

This did occur once, several months ago, but it was short-lived. I updated PHP from 5.5.8 on Friday to 5.5.9, but the issue did not occur immediately, the issue occurred after a restart for something else on Saturday and rolling back to 5.5.8 did not resolve the problem. Right now, with Opcache enabled, we are getting 5-10 failures a day.

[dstogov]

I suppose it's related to some Windows security update. The problem that WinAPI doesn't provide ability to map shared memory into the same memory region. Even worse, Windows performs memory mapping randomization (for security purposes) that complicates things even more.

On Fri, Feb 21, 2014 at 12:33 AM, Keith Davis notifications@github.comwrote:

3 application pools are normally in use. My FastCGI is configured exactly per the instructions in that URL (used those to actually configure the settings.)

This did occur once, several months ago, but it was short-lived. I updated PHP from 5.5.8 on Friday to 5.5.9, but the issue did not occur immediately, the issue occurred after a restart for something else on Saturday and rolling back to 5.5.8 did not resolve the problem. Right now, with Opcache enabled, we are getting 5-10 failures a day.

— Reply to this email directly or view it on GitHubhttps://github.com/zendtech/ZendOptimizerPlus/issues/167#issuecomment-35665676 .

[TerryE]

ASLR seems to viewed as an import security measure to frustrate a range of attacks. It is a pain if you want to create absolute-position SMA, but significantly less so on Windows 64bit because the increased address space reduces the chance of collision with a previously allocated memory region. I came across this interesting postgresql patch on doing a search for ideas here. However we haven't got to stable production 64bit builds on Windows yet.

With 32-bit processes and the 2 Gb memory limit (up to 4Gb on some build options), plus the fact that OPcache as a zend extension basically does its startup phase after all DLLs and PHP extensions have initialised means that the process space is already fragmented with ASLR, and it is therefore difficult to guarantee any absolute address window being available by the point within the process startup chain that shared_alloc_win32.c:create_segments() is executed.

I believe ASLR is now the default, but In the immediate term, the obvious thing to do is to build PHP with ASLR disabled (including all dynamically loaded extensions that you use). Though some security audits might have a problem with this. In the longer term -- probably a PHP6 issue unfortunately -- the sensible thing to do is to make the op_array format position independent. This would have minimal is any runtime hit and would dramatically simplify the whole process of compiled script caching. In fact Dmitry's 5.4 performance enhancements have already laid the groundwork for this.

[laurinkeithdavis]

Due to this issue, we are unable to use the opcache and have had it disabled for about a month. Is this going to get addressed? Is there anything that I can do to help this along?

[TerryE]

@laurin1, did you read and understand my post? In short, ASLR doesn't play well with absolute addressed SMA. However, the current OPcache architecture uses absolute addressing for inter-element referencing in the compiled scripts that it stores in the OPcache SMA.

System-wide disabling of ASLR is the simplest work-around, but this is again best security practice on Windows because of the number of Windows OS attacks which exploit specific DLLs being mapped at known memory locations.

On 32 bit systems where the occupancy of the address space is quite dense, the use of ASLR means that some PHP process spaces will invariably already have previously loaded DLL blocking the putative SMA and PHP initialisation will fail.

This one isn't going to be fixed any time soon IMO. There is no "just add this simple patch" for this one. We'd need to move to position-independent compiled script format. I know how to do this as I use such a format for my MLC OPcache demonstrator, but doing so won't happen until 5.7 at the absolute earliest, even if view as acceptable by the Zend team.

[laurinkeithdavis]

I did see and (mostly) understood it. So, that leaves us on Windows and IIS without an opcode cache solution right now? Wincache is dropping support for their opcode cache feature soon (and we've had trouble with the current version of the opcode cache in Wincache anyway.)

[TerryE]

As far as I see it: disabling ASLR system-wide is the simplest workaround. You can do this on a per process basis but you'd also need to disable it for the DLLs that the PHP image maps in. I can't think of a simple workaround. Maybe I should raise this one on Internals.

[mattficken]

A simple way to fix this issue would be to regenerate the opcache if it can't map the file or find a free memory address (this was the fix I suggested over a year ago).

This doesn't fix the underlying problem (differences in Windows & posix shared memory architectures), but it will allow PHP to function on the rare occassion when this occurs. Performance: shouldn't decrease performance the rest of the time, and allows >0 performance when it does occur.

Wincache now does an equivalent of this for its mapped files (it doesn't search for addresses) as Wincache also can have this issue (for its opcache or filecache).

See: http://svn.php.net/viewvc/?view=revision&revision=333880 https://bugs.php.net/bug.php?id=67466

Really seems like this would be a good way to fix Opcache on Windows NOW, so users can count on it always working.

What does everybody else think?

BTW, issue #109 is a duplicate of this issue.

[laurinkeithdavis]

I don't know how hard or performance adverse that solution is, but I can agree that at least is a solution. We can't use Opcache right now at all, haven't been since for months, and won't be able to until this issue is fixed.

[TerryE]

@laurin1, @mattficken, as I said above the root issue here is the combination of MS and PHP architectures. ASLR is a sensible security feature and it works fine with absolute addressing as long as your OS supports a proper clone-style process forking which is the case for Unix and Linux variants, but this just doesn't work well for current Windows. I've been meaning to discuss with Dmitry the idea of moving to a PIC opcode format for phpng, but I've been very busy with other things of late. This would allow PHP to support windows properly.

The real issue with matt's suggestion is that if the first process is created with a workable address range for the cache, then we could take a 5%, say, failure rate for extra processes with some retry logic. But if the first process happens to map a fluke window that is unlikely to come up again, then your failure rate could be 95% or worse.

This is really a 32 bit address space issue, because with 64-bit address spaces you are dropping your boat in the Pacific ocean and the probabilities of collision are so low that a simple retry logic would work. But 64bit support for PHP is still experimental, and the 32 bit address space is getting very cluttered.

This is not a trivial one to fix because of the clash of PHP's need for an absolutely addresses shared window and MS's current architecture really going out of its way to make this pretty impossible to guarantee.

[LukeMauldin]

According to this Zend bug report: http://kb.zend.com/zend-enabler-error-could-not-locate-a-free-php-process/#.VL7TBf3F9eZ this issue was resolved in Zend Server 4.0.6. Does anyone know if this fix was ported to this extension?

[dstogov]

There's nothing to fix. You can use opcache.mmap_base workaround out of the box.

On Wed, Jan 21, 2015 at 1:19 AM, LukeMauldin notifications@github.com wrote:

According to this Zend bug report: http://kb.zend.com/zend-enabler-error-could-not-locate-a-free-php-process/#.VL7TBf3F9eZ this issue was resolved in Zend Server 4.0.6. Does anyone know if this fix was ported to this extension?

— Reply to this email directly or view it on GitHub https://github.com/zendtech/ZendOptimizerPlus/issues/167#issuecomment-70746167 .

[LukeMauldin]

Is the opcache.mmap_base a permeant fix? Based on my understanding of the issue, it could reoccur randomly due to the ASLR implementation in Windows.

[dstogov]

ASLR may always make problem :(

On Wed, Jan 21, 2015 at 5:00 PM, LukeMauldin notifications@github.com wrote:

Is the opcache.mmap_base a permeant fix? Based on my understanding of the issue, it could reoccur randomly due to the ASLR implementation in Windows.

— Reply to this email directly or view it on GitHub https://github.com/zendtech/ZendOptimizerPlus/issues/167#issuecomment-70841305 .

[LukeMauldin]

Since ASLR may always cause a problem, how does setting the mmap_base address help? We are currently running php-5.4.4 and we rarely get this error but when testing an upgrade to php-5.6.4, we encounter the error more often. Do different PHP versions impact the probability that the error will occur?

[ganeshkbhat]

https://github.com/zendtech/ZendOptimizerPlus/issues/109

This ERROR 500 - Server Error (End of script output before headers: php-cgi.exe) happens with general small scripts as well when using 5.4.36 and Apache 2.4. I just couldnt run PHP scripts. Disabling opcache did the work for me in PHP 5.4.36's .ini file . Have tried increasing memsize usage and other options but did not help. Is this a unresolved issue?

[fedevegili]

I was having a related issue and found this thread.

My IIS (7.5) wasn't loading any page, it was showing a FastCgiModule fatal error, 0xfffffffe.

On windows event visualizer, Zend was reporting:

Unable to write base address and C:\Windows\TEMP\ZendOPcache.MemoryBase@SE Access denied.

The problem was that zend didn't have full control on folder C:/temp. I just set permission to everyone and everything worked.

I'm just posting this here because I didn't find anything about this error on google and it may help somebody else.