Thread safety issue when loading dynamic modules on Linux

Observed while using crate libloading

Given a multi-threaded test that loads a dynamic module, calls a function, and then unloads the module in a loop, the expected behavior is for the module to be loaded/unloaded and it's init/fini called respectively everytime. Like so:

test start
init
dynamic function
fini
init
dynamic function
fini
test end

The expected behavior is observed when running on MacOS but not on Linux.

On Linux we observe the module's init/deinit function being called at the beginning and end of the test but also sporadically (but always in order) throughout the test. This hints that modules are not being unloaded each time (and instead shared amongst threads).

test start
init
dynamic function
.....
fini
init
dynamic function
fini
init
dynamic function
...
fini
test end

In addition to the behavior difference, on Linux this behavior also mostly results in a segmentation fault at some point in the program's execution.

No workaround to the behavior differences is known, but there is a workaround to the segmentation fault. Using RTLD_NODELETE when loading the module forces the loader to not unload the module until the process is killed (init called at beginning of test and fini at the very end).

https://github.com/solana-labs/solana/blob/dffb77bf55c2ff8634bc82c9d791257c6072de21/src/dynamic_program.rs#L78

The following repository will recreate the issue: https://github.com/jackcmay/dynamic_module_segfault

This issue is not to be confused with another loading issue on Linux having to do with TLS:

solana-labs / solana

Thread safety issue when loading dynamic modules on Linux #1314