apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.58k stars 1.32k forks source link

Investigate dlmopen+LM_ID_NEWLM vs copying libfdb_c.so multiple times #6300

Open sfc-gh-almiller opened 2 years ago

sfc-gh-almiller commented 2 years ago

To enable having multiple network threads, the current client implementation opens the shared library multiple times. For these to be given separate instances of static data, one copy of libfdb_c.so is created per thread, so that it appears to be and is loaded as a "different" instance of the library for each thread.

There's an alternative approach to this, which is to utilize the linker's notion of namespaces, to instead dlmopen() a new instance of the library into a new namespace for each thread. This removes the need to copy libfdb_c.so, and to have a secure temporary location for the client library files. This will add a limit of a maximum of 15 network threads (as there's a 16 namespace limit in glibc), and potentially the loading will be tricker as any dynamic dependencies will need to be satisfied manually. But avoiding the copy seems nice for both startup time and security, so possibly still worth investigating.

sfc-gh-vgasiunas commented 2 years ago

Interesting, I was not aware of this feature. Thanks for bringing it to our attention. Probably it could save some loading time and space in memory. I am however concerned about the namespace limit. 16 is a quite hard limit. For example if we use MVC with 2 versions of a client, then we have only 8 namespaces per version, or we have to reuse the namespaces somehow. Also I see the default limit is 4 namespaces and we need to change the glibc tunables:

https://www.gnu.org/software/libc/manual/html_node/Dynamic-Linking-Tunables.html