jmikola
commented
2 years ago When FPM spawns a PHP worker process, PHP invokes the MINIT function of each enabled extension. For ext-mongodb, PHP_MINIT_FUNCTION(mongodb) will be called. If we step through that and ignore the PHP-specific stuff (e.g. registering classes), we see two notable actions:
- Calling
bson_mem_set_vtableto instruct libbson and libmongoc to use PHP's persistent memory allocation functions - Calling
mongoc_initto initialize libmongoc and its dependencies
Diving into libmongoc, mongoc_init delegates to _mongoc_do_init with logic to ensure that function is only called once per process. When compiled against OpenSSL, _mongoc_do_init will call _mongoc_openssl_init to initialize OpenSSL. I'll stop there for now, but let's remember that function as a future point of investigation for the various OpenSSL function calls that may conflict with whatever else is happening in your environment.
The debug log you provided is unfortunately not helpful here, as it's only reporting tracing from some of libmongoc's own function calls. Since you're not interacting with the driver in this request, the debug log is only reporting some functions that get called from mongoc_init. In this case, we see some trace logs from functions in mongoc-linux-distro-scanner.c, which is called via mongoc-handshake.c (when _mongoc_do_init invokes _mongoc_handshake_init).
To investigate this further, I'll need a GDB backtrace of the segfault.
In my debugging, i wrote a simple test script that generates a JWT token using the same library.
What does "same library" refer to in this case? I assume the "library" is a PHP package that interacts directly with ext-openssl, but correct me if I'm mistaken.
If you're able to share that "simple test script" that also produces a segfault when run through FPM, that'd be most helpful. Sharing composer.json and composer.lock (if applicable) might also help.
I develop in docker and i'm unable to reproduce the issue in my development container however i can still reproduce it in my actual servers.
What are the differences in software versions between your production environment and Docker container?
dadamssg
remicollet
Bug Report
I have 3 servers running php with the mongo extensions installed. On Feb 13, all of them encountered the same error at the same time. The error being a segfault coming from libcrypto.so.1.0.2k.
The odd part(aside from all the servers experiencing the same thing at the same time) is that the code i'm running is not even using the mongo extension. We use JWT tokens for auth. This crash seems to come up only on api requests that are interacting with a crypto algorithm, to either encode or decode a JWT token. I disabled the mongo extension, restarted php-fpm, and things started working normally again. Thankfully i'm just using mongo to store xhprof profiles on demand but i have no idea what changed on all 3 of my servers at the same time to cause this.
In my debugging, i wrote a simple test script that generates a JWT token using the same library. i was able to run this script fine from the command line with the mongo extension enabled. It appears a magic combination of the following causes the error:
I was able to make requests against my public api endpoints(that don't require any crypto) with the mongo extension enabled and those responded fine.
I develop in docker and i'm unable to reproduce the issue in my development container however i can still reproduce it in my actual servers.
Environment
Expected and Actual Behavior
Debug Log