Provide mmap implementation

[dmik]

The current mmap implementation lives in a separate dll (mmap.dll) and uses WPSTK which provides very limited functionality for memory mapped files. In particular, it doesn't support MAP_SHARED and working with mapped regions from other threads. See also http://trac.netlabs.org/ports/ticket/32.

The WPSTK dependency is about to be dropped by providing a fresh mmap implementation in LIBCx.

[dmik]

Note that as a side feature, when integrating the LIBCx exception handler to kLIBC that will serve mmap memory access, I also added automatic installation of the EXCEPTQ trap report generator to the main and all other threads created by _beginthread. This allows to get these nice trap reports just by linking your application against LIBCx DLL and saves you from manual EXCEPTQ registration (which may be kinda tricky if the app has many entry points and creates a lot of threads in different places). Note that this registration is dynamic i.e. neither LIBCx DLL nor the app needs to be statically linked to the EXCEPTQ DLL.

Perhaps we should provide the user an ability to disable automatic EXEPTQ installation by setting an environment variable; this is to be discussed.

[dmik]

The initial mmap implementation I added above only deals with MAP_SHARED|MAP_ANON case, the rest will be added in a while. The anon_mmap.c test case from Paul Smedley from this ticket http://trac.netlabs.org/ports/ticket/32 works like a charm now.

There are the following limitations though:

• munmap() presently only unmaps whole mappings (i.e. when addr equals to the value returned by mmap() and len equals to what was passed to mmap()). POSIX requires munmap() to be able to unmap multiple mappings at once s well as unmap partially (might even result into two new mappings — when the unmapped region is in the middle of the mapping). While uncapping multiple mappings can be easily implemented, partial unmapping isn't that easy as OS/2 doesn't support freeing memory partially. For such cases munmap() silently returns 0 (kinda POSIX requirement as well).
• MAP_SHARED mappings are only inherited by fork()ed children and grandchildren. Children created with spawn() won't get access to those mappings. This is done so that POSIX explicitly mentions that MAP_SHARED is only inherited by fork(). We could add inheritance to spawn() on OS/2 as well but that will require the parent to pass the mmap address to the child through some means which makes it not very useful. More over, there is also POSIX shm_open API which is much like DosSharedMem and fits this task better.

[dmik]

Also I get some dirty LIBC crashes when I change tst-anon_mmap.c so that the child also forks another child. I need to debug that, this may be related to parent/child fork callbacks LIBCx installs now.

[dmik]

Note that the rest of mmap functionality (e.g. MAP_PRIVATE and non-anonymous mappings) is still forwarded to mmap.dll (based on WPSTK) for the time being. This forwarding (and mmap.dll dependency for libcx0.dll) will be removed once this issue is closed.

[psmedley]

Thanks dmitry, I'll build the opcache module for PHP and see if it works with this implementation :)

[psmedley]

Unfortunately opcache isn't working. PHP configure uses the code in http://smedley.id.au/tmp/map_anon_test.c to detect anon mmap support - the configure test returns rc = 5.

If I hack php_config.h to force map_anon - opcache crashes php at startup, exceptq report at http://smedley.id.au/tmp/BAB4_01.TRP

[dmik]

@psmedley thanks for testing, Paul. Rc = 5 means that you most likely have -Zno-fork in your CFLAGS (which effectively disables fork). You must never use -Zno-fork with your kLIBC apps (unless you absolutely have to, but I'm not aware of such situations, not when you compile Unix/Linux software at least).

[dmik]

That said, the above map_anon_test.c works great here with the latest libcx0.dll build (returns 0 as it should).

[psmedley]

from PHP's config.log - no -Zno-fork here.... configure:65471: checking for mmap() using MAP_ANON shared memory support configure:65526: gcc -o conftest.exe -I/usr/include -DEMX -DEMX -DOS2 -DST_MT_ERRNO -O2 -march=pentium -mtune=pentium4 -static-libgcc -L/usr/lib -s -Zexe -Zomf -Zmap -Zhigh-mem -Zstack 8000 -DST_MT_ERRNO -static-libgcc -L/extras/lib -LU:/extras/lib conftest.c -lcrypto -lssl -lcrypto -lz -lm -L/openldap/lib -lpthread -lldap -llber -lssl -lcrypto -lz -lexpat -lssh2 -lgcrypt -lgpg-error -lpoll -lbz2 -llzma /extras/lib/iconv.a -llibcx0 -lxml2 -lz -lm -lpoll -lxml2 -lz -lm -lpoll >&5 configure:65526: $? = 0 configure:65526: ./conftest.exe configure:65526: $? = 5 configure: program exited with status 5

[psmedley]

Is it possibly something related to your recent libc066.dll fixes? I'm using Knut's libc066 here.

[dmik]

There is such a possibility, fork is kinda broken in libc066. Please try libc066.dll from the latest RPM.

[psmedley]

Ok I will test with rpm libc066 asap - hopefully this afternoon, depends how much wine I drink at lunch :)

Cheers,

Paul

On 27 Aug 2016 09:31, "Dmitriy Kuminov" notifications@github.com wrote:

There is such a possibility, fork is kinda broken in libc066. Please try libc066.dll from the latest RPM.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitwiseworks/libcx/issues/11#issuecomment-242880514, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-ccvtzz7a8iboToghBj4Ph8Ql25llcks5qj35cgaJpZM4JorAS .

[psmedley]

config.log output with rpm libc066 is better: configure:65471: checking for mmap() using MAP_ANON shared memory support configure:65526: gcc -o conftest.exe -I/usr/include -DEMX -DEMX -DOS2 -DST_MT_ERRNO -O2 -march=pentium -mtune=pentium4 -static-libgcc -L/usr/lib -s -Zexe -Zomf -Zmap -Zhigh-mem -Zstack 8000 -DST_MT_ERRNO -static-libgcc -L/extras/lib -LU:/extras/lib conftest.c -lcrypto -lssl -lcrypto -lz -lm -L/openldap/lib -lpthread -lldap -llber -lssl -lcrypto -lz -lexpat -lssh2 -lgcrypt -lgpg-error -lpoll -lbz2 -llzma /extras/lib/iconv.a -llibcx0 -lxml2 -lz -lm -lpoll -lxml2 -lz -lm -lpoll >&5 configure:65526: $? = 0

Will rebuilt PHP 5.5.x with opcache now :)

[dmik]

Btw you should remove -lpoll when you use -llibcx0 as the latter provides a better version of the former. Also please link to LIBCx as -lcx (there is an a.out lib file provided and you should avoid version suffix as it may change).

[dmik]

Btw2, I see this comment from you in notifications but not here:

Unfortunately, php still crashes at startup with the latest libc066, exceptq log attached. I should probably install some xqs files to improve the log..

Did you delete it? Anyway, the .dbg symbols for the new libc066 are in the RPM/ZIP in the repo.

[psmedley]

Hi Dmitry,

On 27/08/16 19:02, Dmitriy Kuminov wrote:

Btw2, I see this comment from you in notifications but not here:

|Unfortunately, php still crashes at startup with the latest libc066, exceptq log attached. I should probably install some xqs files to improve the log.. |

Did you delete it? Anyway, the .dbg symbols for the new libc066 are in the RPM/ZIP in the repo. Yes I deleted it as I realised that apache2 was crashing with or without opcache loaded - ie mmap likely isn't the issue... more to come...

[psmedley]

OK - I still can't get mmap working with either the opcache plugin for PHP 5.5.x, or with PHP 7.x which requires mmap (or equivalent code on win32) to run.

exceptq traps at https://dl.dropboxusercontent.com/u/76425158/mmap_trp.ZIP (github wouldn't let me upload a zip)

[psmedley]

mmap example code at https://www.safaribooksonline.com/library/view/linux-system-programming/0596009585/ch04s03.html fails with 'Permission denied' - is this expected? the mmap call is p = mmap (0, sb.st_size, PROT_READ, MAP_SHARED, fd, 0);

[dmik]

Yes this is kinda expected. The MAP_SHARED and not MAP_ANON case is not yet implemented. Why did you use this example? Does opcahe use this scenario? Anyway I need to implement it too.

Regarding the TRP files. You could upload them here directly instead of a ZIP - this is more convenient. It looks like the trap happens because of non-implemented mmap cases but that needs to be checked in the sources. I will also give you the debug LIBCx dll for more logging (first I need to fix another major problem in LIBC related to fork).

[psmedley]

I was looking for example code which gave similar results to PHP failures.

PHP 7.0.x uses ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); so this is expected to not work yet.

opcache uses: shared_segment->p = mmap(0, requested_size, PROT_READ | PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0); which I guess should work?

btw github wouldn't allow me to upload TRP files for some reason...

[dmik]

Your guess is right. Anyway, hold on, the complete impl is on its way.

Regarding TRP files, how do you attach them?

[dmik]

In 9bfacaa1c304469f40e390c4a56499a81801f7f8 I added the MAP_PRIATE|MAP_ANON implementation so now it should cover all PHP needs, both PHP itself and opcache.

@psmedley could you please test it again? The new DLL/LIB is uploaded as http://rpm.netlabs.org/test/tlx2.zip. And the debug version generating many logs (in case if you run into trouble) is available there as tlx2d.zip.

In addition to limitations mentioned above there is also one more minor thing to improve:

• Currently, internal shared mapping structures won't ever be released (until the last LIBCx application terminates, at least) if the process allocating it or gaining access to it due to fork() doesn't call munmap. Shared memory itself will be in fact released by OS/2 when all processes having access to it are terminated. However, if there is some other LIBCx process not having access to the given shared map still hanging around, there may be unexpected result if the same memory address gets reused by this process for some other means (because the LIBCx mmap exception handler will kick in for the given address range in such a case and it may try to unexpectedly commit the memory).

This needs a proper fix. I plan to do it now and then I will switch to file-based map files.

[dmik]

I implemented releasing shared mmap regions for which munmap wasn't called at process termination and their further deletion when the usage counter drops to zero in 0de66d22b874f17abdf89b7ffef211029cd691d2. I found a neat solution that doesn't require to associate a list of PIDs with each shared mapping in order to track which process didn't call munmap at termination. As unmapping always frees memory in the process address space, the respective pages become marked with PAG_FREE if munmap was called and remain allocated/committed otherwise by the time when the process is terminated. And this is a perfect indicator of whether to decrement the usage counter of a shared mapping at process termination or not.

[dmik]

@psmedley Please forget about the tlx2 builds and use tlx3 instead.

[psmedley]

@dmik I did some testing this evening. First I tried with self-compiled libcx and things didn't work. I then installed the logging libcx0. Logs uploaded as well as the matching exceptq logs

php7libcx0.txt php7trp.txt opcache-libcxlog.txt opcache-trp.txt

[dmik]

Thanks, Paul. Regarding PHP, according to the logs, it's calling mmap only with one flag - MAP_ANON. This is illegal, a mmap call must also contain either MAP_PRIVATE or MAP_SHARED. Hence errno 22 (EINVAL). A subsequent crash most likely happens because PHP doesn't handle the mmap error properly and tries to use it anyway.

Regarding Opcache, it looks like for some reason the LIBCx exception handler is not called. May be apache messes up with the exception handlers... Does Apache/PHP install EXCEPTQ on its own?

BTW, did you use the latest version of LIBCx in the above tests, e.g. tlx3?

[psmedley]

php7-exceptq.txt Latest PHP7 testcase log attached. Using http://smedley.id.au/tmp/php7test.zip should allow a local test. I'm just running 'php.exe test.php' php7-libcx0.txt

@dmik commented on IRC 're PHP, it wants to do partial region unmapping, we currently don't support that'

[dmik]

A second look on the last PHP7 log gives me a thought that partial uncapping is not the cause of the crash (it returns a failure but PHP7 seems to ignore it). The TRP report shows that PHP tries to access a structure pointer which is NULL. Paul, can it be some other test leftover in the source?

[psmedley]

OK, opcache appears to work with PHP v5.5.x - at least some simple pages load.

re: PHP7 - I double checked the diff's I don't believe there are any stray leftover tests in the source.

[dmik]

Okay, good. Re PHP7, does it work well with mmap disabled? If so, than this crash is somehow an indirect result of some difference in the mmap implementation. Or some missing bit in the OS/2 port compared to PHP 5.5. I can reproduce the crash locally (which is good) but I can't debug it w/o PHP sources... Can you please track down the exact code path which fails (at source level), usingprintf tracing or such? This would be of great help.

[psmedley]

PHP7 requires mmap, PHP5 didn't. The previous mmap implementation didn't work with PHP7. I'll work on tracking down the code path for the crash

[psmedley]

re: php7 I think you're right that it has nothing to do with mmap. It is something related to initialisation of zend extensions - will investigate more as I get time, but for now, I don't believe that it's mmap related.

[psmedley]

re: opcache - whilst some pages load, others fail. Apache2 generates some TRP files. I've created a zip with a selection of TRP files, along with the libcx0 log from the tx4d.zip OPCACHE.ZIP

[dmik]

re opcache, the TRP may be safely ignored — they are from EXCEPTQ you install manually from within Apache. You should really remove this manual EXCEPTQ installation as it just mixes things up (as long as you link against LIBCx, you will get EXCEPTQ reports anyway). And from what I see in the log file, LIBCx works quite well. At some point it starts getting DOS error 8 from DosAllocSharedMem in response to mmap (which returns a failure with errno set to ENOMEM in such a case). DOS error 8 mean just the same: ERROR_NOT_ENOUGH_MEMORY. Which means that greedy opcache wants to mmap more than OS/2 can offer it (and may be doesn't handle failures properly). So either try to manipulate with VIRTUALADDRESSLIMIT in config.sys to increase the address space or you need to patch opcache so that it doesn't try to eat more than OS/2 can offer. Anyway, this doesn't look like there is a problem in LIBCx at this point.

[dmik]

In the meantime, in 821b375ae1d6ae13c4656bdb9ae0d3b7fac26a02 I implemented MAP_PRIVATE mappings for regular files which drops the WPSTK (and mmap.dll) dependency. There are still some missing bits compared to mmap.dll like msync and mcommit but for MAP_PRIVATE using msync makes no sense anyway (and its implementation in mmap.dll looks wrong to me) and mcommit looks like an OS/2 specific extension that should not appear in Unix code. That said, I think LIBCx may be already used in place of mmap.dll in all our Unix ports with the same level of functionality.

Technically, MAP_SHARED for regular files should also work now but I didn't add code to synchronize memory writes with the underlying file contents yet (i.e. write out pages changed in memory back to the file). This is my next task along with masync and also mprotect (which is also frequently used in the Unix world and which we had to change to DosAllocMem API in many ported apps back then).

[LewisR]

As the thrust of this mmap stuff ATM is for PHP, I figured I'd jump in here with some testing observations running PHP 5.5.36 under Apache 2.2.29 (Paul please confirm that the php module is compatible with 2.2, and that we don't need a recompile vs 2.4).

Result is that PHP does load under Apache in the test environment. PHP info page is accessible, and shows that we are caching. opcache log reports objects being added, and no errors are logged in php.err. Unfortunately,. pages do not render (throbber continues to spin, no Apache or PHP crashes observed, but pages are not returned).

opcache settings in php.ini (highly speculative; remainder of options are unset/unchanged from default):

opcache.memory_consumption = 128
opcache.interned_strings_buffer = 16
opcache.max_accelerated_files = 3281
opcache.max_wasted_percentage = 25
opcache.use_cwd = 1
opcache.validate_timestamps = 1
opcache.revalidate_freq = 30
opcache.revalidate_path = 1
pcache.save_comments = 1
opcache.load_comments = 1
opcache.fast_shutdown = 1
opcache.enable_file_override = 1
opcache.dups_fix = 0
opcache.max_file_size = 0
opcache.force_restart_timeout = 120
opcache.error_log = 'j:\apps\php5\logs\opcache.log'
opcache.log_verbosity_level = 4
opcache.preferred_memory_model = mmap

I am seeing a collection of the following in %TMP%:

9-03-16   1:56               0    124   ___A_  .ZendSem.0F2VMi
9-03-16   1:56               0    124   ___A_  .ZendSem.0GfA7a
9-03-16   1:56               0    124   ___A_  .ZendSem.6BWtrc
9-03-16   1:57               0    124   ___A_  .ZendSem.6Fz86z
9-03-16   1:56               0    124   ___A_  .ZendSem.6uKH6D

Whether any of this is directly related to mmap or libcx vs opcache, its configuration, or PHP, I have no idea.

[psmedley]

@dmik Re: exceptq - I thought I was running an apache2 build with exceptq support removed, I'll need to confirm this evening, and also check PHP to see if that has exceptq support included that should now be provided by libcx0.

@LewisR re: apache2 module - the one I linked to is for apache 2.2 only, it would need to be rebuilt for apache 2.4

[psmedley]

@LewisR There is a reference to '#define SEM_FILENAME_PREFIX ".ZendSem."' in the opcache source - I'll do more digging when I get some time.

@dmik you're right that exceptq was still enabled in apache2 - when I disable it, there are no TRP files, but apache2 error log shows: [Mon Sep 05 18:59:52 2016] [error] caught exception (XCPT_ACCESS_VIOLATION) in worker thread, initiating child shutdown pid=4657 tid=3

Looks like something in opcache is crashing and pages don't return. Will try work it out....

[dmik]

@psmedley hmm, this means Apache installs its own exception handler... I really wonder if it interferes with mmap or not. A badly written one can certainly do so. Can you point me to the code where this message originates from?

[dmik]

@LewisR yes, I agree that we should test PHP 5.5 and PHP 7 separately to isolate PHP 7-specific and mmap-unrelated problems.

[dmik]

In the mean time, in 090efa0d105188abc223a788823391689d55a14f I implemented MAP_SHARED support for regular files. Its current limitation is that the changed bytes are flushed back to the underlying file only when munmap is called. In theory a proper mmap implementation should also flush dirty pages at some reasonable intervals before munmap, e.g. to make sure that most of the changes are preserved if a power failure occurs. This looks quite important for me so I have to implement it.

Regarding the msync implementation. Since you can't really control the async behavior of DosWrite (and there is no Async I/O per se on OS/2), I plan to implement it by simply requesting an immediate flush (so that it acts like the MS_SYNC flag is given on input because this assumes data ends up in a file immediately from a subsequent DosRead POV at least). And the MS_ASYNC flag will be silently ignored as well as MS_INVALIDATE which doesn't apply to OS/2 at all if I read it right).

[dmik]

There is also a request from @LewisR for madvise here: http://trac.netlabs.org/ports/ticket/20. The POSIX version of it has the following spec: http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_madvise.html. And here is the Linux version (pretty much similar): http://man7.org/linux/man-pages/man2/madvise.2.html. I agree with @ydario that both can be simply implemented as stubs on OS/2 since the OS/2 kernel doesn't really take any advises from the application about memory usage other than shared/private and protection flags. I will do that within this ticket as well.

[psmedley]

@dmik apache2 errors comes from http://trac.netlabs.org/ports/browser/apache2/trunk/server/mpm/mpmt_os2/mpmt_os2_child.c - I forgot - apache2 does have it's own exception handler.... this code goes back to original apache 2.0 builds from Brian Havard....

[dmik]

@psmedley looks like my guess was right - this own exception handler doesn't behave correctly as it deliberately terminates the process upon XCPT_ACCESS_VIOLATION w/o letting other exception handlers in the chain act (in this particular case - the mmap exception handler from LIBCx). This is wrong and a well written application must never do that. Anyway, this exception handler looks completely redundant to me — the process will be terminated by OS/2 anyway if there is no handler to recover from a given exception. And given that this own handler is disabled when EXCEPTQ support is on (and Apache somehow works w/o it), I think we should simply drop it at all.

Judging from the code, the only useful thing this exception handler ever does is this:

        for (c=0; c<HARD_THREAD_LIMIT; c++) {
            if (ap_scoreboard_image->servers[child_slot][c].tid == _gettid()) {
                ap_scoreboard_image->servers[child_slot][c].status = SERVER_DEAD;
                break;
            }
        } // for

This basically sets an indicator that the process has died abnormally. The best place to do such a thing (if it's really needed which I doubt since USE_EXCEPTQ completely disables this code anyway) is in the process exit handler registered with DosExitList (this handler will have enough information for that), no exception handler installation is necessary.

So please disable the whole thread_exception_handler thingy and all its usage for now and test again.

[dmik]

Fixed a bunch of small glitches with MAP_SHARED and added sys/mman.h to the repo with the necessary fixes (it will replace the header supplied with kLIBC).

[dmik]

BTW, mprotect is actually already in kLIBC (looks like Knut needed it for some of his own needs) and given that its function isn't actually related to mapped files per se, the existing implementation is quite acceptable (it's basically a POSIX interface to DosSetMem). So what's left is a worker thread that flushes to files periodically, msync that causes an immediate flush and the madvise stub.

[psmedley]

OK, apache2 exception handler disabled in http://smedley.id.au/tmp/httpd-20160908.zip Logs now show: Fatal Error Unable to allocate shared memory segment of 134217728 bytes: mmap: Not enough memory (12)

I'll experiment more with virtualaddresslimit - current value is 1536 - VM has 2gb of ram allocated.

[psmedley]

NOTE current git source fails to compile with: U:/DEV/libcx0-git/src/mmap/mmap.c:42:48: error: 'PAGE_SIZE' undeclared (first use in this function)

[dmik]

There are problems with the flush thread. First, it is bound to the process and if this process terminates while other LIBCx processes using mmap are still up, a new thread must be created in one of these processes by request. This isn't an easy task. We can pre-create a sleeping thread in each process to overcome this but I don't like this solution much.

Another problem is thread shutdown. By the time LIBCx kicks in at process termination, all threads but thread 1 area already killed by OS/2. This includes the flush thread. So if this thread is in the middle of the file write operation, the file sync will be incomplete. Not good at all.

[StevenLevine]

Sounds like there's a need for a flush daemon that does everything possible to avoid dieing when it should not. It could keep track of the other mmap users and do the right thing when one of them dies unexpectedly.