Request: Multi-threaded I/O

[pixelherodev]

I have a program that launches a group of Lua-based plugins. The plugins are instantiated in parallel; that is, there are a number of threads that set up the plugins.

However, I came upon an assertion failure when using IO::Load to load in the Lua scripts, as apparently IO::Load can only be called from the thread IO::Setup was called from? Is there any way to change this so that I/O can execute on other threads?

[floooh]

Embarrassingly, this doesn't appear to be possible, and a fix isn't trivial (definitely not as easy as making a few classes thread-safe).

As far as I can see at the moment, the tricky part is that the completion callback would need to be called from the thread that initiated the IO, currently the completion callback is always called from the main thread, so the IO system needs some sort of thread-local context and runloop for checking for IO completion.

Definitely not something that can be fixed with a few lines of code unfortunately, but I'll keep the ticket open, as this is definitely something that should be fixed in the future.

[floooh]

Notes to self:

• the IO system needs to maintain per-thread loadQueue's (since the loadQueue manages the completion callbacks)
• ioWorker::put() must be callable from any thread (access to writeQueue needs to be thread-safe)

[pixelherodev]

I might see if I can implement this for you. I would definitely enjoy having it :P

[danhambleton]

I'm running in to this as well. It would be great to be able to call IO from any thread. However, in my case, an acceptable solution would be a single threaded load (i.e. fully resolve the resource load on any thread or even a specified background thread). I wonder how hard it would be to do something like IO::LoadNow()?

[floooh]

If the IO::LoadNow() would basically be a synchronous load which finishes before the function returns, that's unfortunately not possible on asm.js/wasm. I'd prefer to not have per-platform features like that.

[danhambleton]

Hrmm, that makes sense :)

I guess I can use IO::ResolveAssigns to get the resource path and implement custom behaviour from there.

[pixelherodev]

It's not possible to do synchronous loading on JS / wasm?

[floooh]

It's not possible to do synchronous loading on JS / wasm?

Nope, the application basically fires an asynchronous XmlHttpRequest (or that new Fetch-API request stuff), and a couple frames later a callback with the result (or error) will be invoked.

[pixelherodev]

I was looking into this further, and apparently synchronous XHRs are allowed on threads other than the main one? If I understood that correctly, would it be possible to add in a synchronous IO::LoadNow that's usable in such contexts? That would allow for that functionality to be exposed on other platforms as well without the API getting out of sync

[pixelherodev]

I'm no longer interested in seeing multi-threaded I/O (though if you are, you can of course leave the issue open)