dlopen in asm.js

[kripken]

The first two testcases work on asm_dlopen2. That means you can build asm.js programs and shared libraries, and basic dlopen/dlsym work. However, the other testcases assumed shared libs could directly access globals in the parent, which is not true natively and doesn't make sense to support in asm either. So I am not sure how much currently works.

Problems

• Returning 64-bit integers (or bigger) across modules will fail
• C++ exceptions are not supported yet

Limitations

• Each module has its own stack. This wastes some space.
• Using compiled libc stuff in a shared library requires that it be exported from the main module (EXPORTED_FUNCTIONS, or EXPORT_ALL).

Help is welcome in writing testcases for stuff not currently working, on or not on that list.

[kripken]

docs on https://github.com/kripken/emscripten/wiki/Linking

[kripken]

Merged to incoming

[kripken]

Keeping this open for C++ exceptions and returning 64-bit or bigger values, the remaining issues.

[inolen]

related https://github.com/kripken/emscripten/issues/1577

[httpdigest]

Since the wiki page about linking states that dynamic linking is not supported with fastcomp, are there any plans on bringing dlopen to fastcomp?

[httpdigest]

Just came across a really great article about linking on qnx http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/sys_arch/dll.html. Even though I do not have a really thorough understanding of how linking works in theory and practice, in order to support real dynamic linking, an asm.js module would need to resemble the "dynamic section" of ELF (or other object file formats).

[kripken]

I've actually been hoping to get to dlopen in fastcomp. I made some progress on Friday. But I'm not sure how long this will take to do, and it's lower-priority than other stuff that might come up. So I can't promise you if/when it'll be ready. But if you aren't in a rush, might wait to see how that goes. (If you feel like helping, that would be even better of course!)

[httpdigest]

Thanks for the info. That's good to hear it's being worked on. I would die to help out on it, because that and threading are the pain points right now for me. Given that I do not have a thorough understanding of both, I been catching up on operating systems theory courses and C/C++ currently (pitty they were not in my curriculum back then :). Therefore I fear I would not be the ideal one to help out on dynamic loading and fastcomp. But I'll dig into it.

[kripken]

Emulated function pointers (so we can add new function pointers later) landed on incoming. Further linking work is going on in a linking branch. So far I have basic relocation working (i.e. global vars and function pointers can have an offset, they aren't at a fixed absolute address, which would have prevented dynamic loading of code when we don't know where its data will be on the heap).

[httpdigest]

Nice work! I can see now how much work that must be (writing a dynamic loader from scratch :) I would go on and begin writing test cases. Especially for the whole number of different link/resolution modes, such as RTLD_LAZY, RTLD_NOW, RTLD_GLOBAL and such.

[kripken]

I think we have such testcases in the suite, but disabled (test_dlfcn_*). I was planning to look at those soon once the code is ready. If you find something important that is missing we can add it. Although, I am not sure we can or want to implement all parts of the dlopen APIs.

[kripken]

the 2 first tests seem to be passing at this stage, on linking branch. still a lot left to do. but mostly small stuff.

[httpdigest]

Those are really good news! Please keep at it. :) A fully working dynamic linking based on dlfcn would enable so many new exciting things on the web, such as compiling more runtime environments of other languages to the web and using plugin/modular frameworks which make heavy use of dynamic loading. I am currently figuring out how JamVM and Classpath are doing all those things in their guts, and having a fully working dlfcn would enable the same exact build process of Classpath (which utilizes dynamic linking for libraries built by Classpath to be dynamically loaded by JamVM) to be used with Emscripten. Additionally, dlfcn would allow the whole JNI mechanics to work and would allow to reuse the whole ecosystem of existing Java libraries, even if they use JNI themselves (as do wrappers for existing native frameworks, such as FFmpeg), which can then be compiled with Emscripten as shared library and loaded and used with Java inside the browser! :) So, from my perspective, dynamic linking is something I am really looking forward to.

[kripken]

Yeah, definitely. Opens up a lot of new options.

Ok, all the core dlopen tests now pass on the linking branches (need both emscripten and fastcomp repos). That means that dlopen should work, as far as the test suite can tell me - might be a good time to test on quakejs and other projects that use dlopen.

I'll work on more general dynamic linking this week.

[httpdigest]

That's cool. When peeking over the git changeset, there are tests that reference a specific libc function from musl which the shared library is using just so that it will not be optimized away when statically linking libc in the main executable. You surely have thought about the possiblity to have libc itself be a shared library now, instead of being statically linked into each main application? I see a great benefit in that, mostly during development, which is that "libc.so" would only have to be downloaded and cached by the browser once and all of the programs and other shared libraries depending on it would become much smaller in size and faster to build. Or maybe building libc as shared library and dynamically linking against it is also possible right now? But I guess for this to work, a shared library must be able to itself declare the list of other shared libraries it depends on, so that a possible runtime dynamic loader will try to resolve them.

[kripken]

Yes, I intend to get shared libraries working, including for system libraries like libc. But so far just dlopen works.

[httpdigest]

One question: How do I build shared libraries? Using -shared does not seem to work and using -o libmylib.so throws a warning that building shared libraries is not supported. Then I had a look at the test_core.py test, which first compiles the library to a .o.js file and then renames that to .so. Doing that, I get a large LLVM object file. I am using the current linking branch. Then, how do I open the built library? When using dlopen("libmylib.so", RTLD_NOW); nodejs says it cannot find "/libmylib.so". Should it work under nodejs?

[kripken]

I'll document this more when it's finished (getting close, dynamic linking commits going in today). But meanwhile, look at what is going on in the dlopen tests - yes, if you want an .so suffix, you need to rename.

You can use incoming now, linking was merged and the rest of the work is straightforward enough to do without a side branch.

[kripken]

This is now considered complete. Testing is very welcome. Docs are at

https://github.com/kripken/emscripten/wiki/Linking

[httpdigest]

That's great. Thank you for implementing it and for the documentation! I have one question regarding the "architecture" of the solution: Is every shared library a separate asm.js module (with the three parameters, stdlib, external and buffer) and invocations of functions provided by a shared library always uses the "external" parameter of the main application's asm.js module to call outside of asm.js into the asm.js code of the shared library's function? If so, I think that this is a very good solution, albeit maybe a bit slower than native asm.js calls inside the same asm.js code. But that is certainly just a matter of JavaScript engine optimization to make those calls as fast as native asm.js calls. Wouldn't that architecture then allow to go further and provide a shared library build of the system libraries? To avoid the quarries with special handling of system libraries, that you mentioned in the Wiki documentation.

[kripken]

Yes, that is how it is implemented.

To do system libraries that way, we would need to support weak linking, which is used in libc, and probably other linking features for libc++. Not sure it's worth it. In practice, you can just make a main module with all the system libraries, and put your project's code in multiple side modules.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.