Develop a fully multithreaded C runtime for pthreads.

[juj]

Several functions in the C runtime are currently effectively singlethreaded, because they carry global process specific state in JS code that can't be thread local. Therefore calls to these runtime functions from pthreads are proxied to the main thread. Migrating to use more musl can avoid this issue. Also, investigate creating a fully asm.js -based virtual filesystem to be used when targeting pthreads, so that C file I/O can be naturally multithreaded as well.

[juj]

See https://github.com/kripken/emscripten/blob/incoming/system/lib/pthread/library_pthread.c#L252 for a list of the functions that are currently proxied.

[juj]

A special note to clarify: this issue is not only performance-related, but this is a problem because this can cause a special kind of deadlocking that native code does not exhibit. If the following conditions are met:

a) the application was not built with --proxy-to-worker mode, and b) the main thread has entered a blocking loop that waits for a thread before exiting the loop using only atomic lock-free techniques, e.g. an atomic compare-and-swap (CAS), and c) the loop does not call to any pthread api functions, nor to certain C runtime functions that the pthreads api considers to be "cancellation points" (see list here http://man7.org/linux/man-pages/man7/pthreads.7.html), and d) at the same time, the worker thread that the main thread waits on, attempts to perform a proxied C runtime operation,

then the main thread and the worker thread will lock up, since they are both waiting for each other to progress: the main thread is spinning (livelocking) to wait until the worker thread flips a bit somewhere, while the worker thread is sleeping to wait for the main thread to perform the proxied operation before it can continue. Conceptually this is a case of a priority inversion failure between two mutexes, even though mutexes are not technically involved here.

The correct solution to this will be to fix up Emscripten C runtime to not require proxying API calls, i.e. developing a fully multithreaded C runtime, which will remove the need for the worker thread to wait for the main thread.

A workaround solution is to audit all instances of blocking atomic loops that an Emscripten main thread can enter, and add a call to a special function emscripten_main_thread_process_queued_calls(); in the body of that loop. This call allows the main thread to assist the worker thread to perform the C runtime call while the main thread is still spinning to wait for the thread, thus avoiding the deadlock.

[kripken]

Regarding point (a), what is the main thing stopping a pthreads application from doing that today? Proxy-to-worker seems like the best solution all around here.

[juj]

Yes, --proxy-to-worker will be one way to work around this, but I have not had time to start implementing it. The reason is that it is quite a bit more complicated, since there's a large amount of handwritten JS code that will need to be lifted from the main JS thread to the main browser thread. In general any piece of JS code that stores data that should be visible to all threads, e.g. the filesystem, needs to live in the main browser thread and not the main JS thread in order to decouple the blocking.

[kripken]

Oh ok, I see. Sounds like we need to keep the main thread mostly as it is now, but to not run user code in it, so it just receives proxying requests.

[kripken]

So basically, we want to move main() to a pthread, but leave everything else as-is?

[juj]

Yeah, probably the cleanest way would be to load the whole compiled .js file in the main browser thread as well, but never call main() in it.

[gouletr]

What's the actual implications of --proxy-to-worker mode? What would it mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call is put in the proxy queue (once per call name, so that it doesn't spam) so that we know where it happens?

[juj]

The intent is that --proxy-to-worker should be as fully transparent to the application as possible and all code should work nicely with it when pthreads is enabled as well (since we can do synchronous postMessaging via the SAB). Currently there will most likely exist a lot of bugs of various small things that may not be proxied correctly, or may not be proxied synchronously at all if SAB is not enabled. We are working on this, testing and comments about your experience are welcome!

2015-09-16 18:04 GMT+03:00 Robert Goulet notifications@github.com:

What's the actual implications of --proxy-to-worker mode? What would it mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call is put in the proxy queue (once per call name, so that it doesn't spam) so that we know where it happens?

— Reply to this email directly or view it on GitHub https://github.com/kripken/emscripten/issues/3495#issuecomment-140769195 .

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[niklasf]

I believe this is still relevant.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[AlexanderJ]

We had a lot of deadlocks due to the way emscripten_main_thread_process_queued_calls handles calls. I now developed a solution which works well for us, by introducing another function emscripten_current_thread_process_proxied_queued_calls which only processes those calls which are related to proxy to main thread. This way we no longer end in deadlocks when e.g. a functionality uses memory allocation/deallocation in a mutex and then emscripten_main_thread_process_queued_calls invokes the same functionality and deadlocks. I would like to push this, but I cannot create a branch in the project and thus cannot push it to create a pull request for it. Any help appreciated.

[sbc100]

The normal procedure with github is that you create fork of your own and then create the PR from there.

[hcldan]

@juj I'm here following some doc trails in the emscripten project. I'm trying to use the new FS driver for StorageFoundation which requires that the FS driver be loaded into a worker (it uses some sync apis).

I see that we are proxying everything to the main browser thread, but that breaks the FS driver because the sync apis are not defined there. Is there any reason why this needs to proxy to the main browser thread and not a dedicated worker? Any tips on how I could redirect this so that FS could live in a worker?

[sbc100]

@ethanalee who is going to be doing some work on the filesystem? Sounds like you guys should maybe coordinate if you are not already?

[ethanalee]

Hi @hcldan . Are you attempting to use this: https://github.com/fivedots/storage-foundation-api-emscripten-fs ? There is also a guide as well: https://github.com/fivedots/storage-foundation-api-porting-tutorial

Please let me know if this helps you. :)

With regards to proxying for our current filesystem: https://emscripten.org/docs/porting/pthreads.html#proxying

[hcldan]

@ethanalee Yes, I am trying to use the SF fs driver. I'm reading the guide but I'm not sure why the example works yet.. as nearly all of the calls I'm seeing in our multithreaded application proxy all FS calls to the main browser thread. Therefore the FS driver must be there, and it does not have the required sync apis.

[hcldan]

https://github.com/emscripten-core/emscripten/issues/13519

[kripken]

Note that @rstz is doing some experimentation with that here: https://github.com/rstz/emscripten-pthreadfs

Yes, it is a problem that we proxy everything to the main thread atm. A temporary solution is to proxy everything to a worker instead. But that requires some hacks atm, and is not a good long-term solution (https://github.com/emscripten-core/emscripten/pull/14666 helps, but it's still not easy). The proper solution will be a new filesystem rewrite that is just being designed now. That should avoid the mandatory proxying, and it should be easy to add SF support on top of that (still proxying to a worker, but in a reasonable way, and with multithreaded caching).

+1 for collaborating here where possible, there will be plenty of work to do.

@ethanalee Maybe we should open a tracking issue for the new filesystem rewrite?

[hcldan]

@kripken Please let me know where I can help, and how I can start.

[ethanalee]

Opened a tracking issue: https://github.com/emscripten-core/emscripten/issues/15041

[kripken]

@hcldan I think for now we should discuss in that tracking issue to make sure we are not missing any requirements. I think that's all until we have a clear design. At that stage perhaps we can write up a list of independent tasks.