Std Main Loop

[ncannasse]

Each platform comes with its own “main event loop” implementation, making it hard to build crossplatform API over it.

We shoul provide a base abstract loop definition that can help this.

Discussion opened on the topic.

[hughsando]

ousado, You might be reading too much into the scope here. Firstly, it is all about "the main thread", specifically as it applies to either: 1) interacting with windowing frameworks, or 2) preventing the command-line program from exiting with unprocessed events (timer, thread, async io)

This is to solve these two very real problems. Solving efficient multi-thread communication is out of the scope of this change, and this does not dictate how you might choose to implement this. It only gives you a standard way of preventing command line apps from exiting and running stuff on the main thread inside someone else's windowing framework.

This code needs the be in the same repo as haxe.Timer and haxe.http and probably cpp.vm.Thread. And since these are in the std lib, this should be too.

Currently we get people who start a timer on the cpp target, but the program exits before the timer executes, and then I get a bug report because this is not how it works in js/flash. Similarly, not waiting for threads to finish surprises some people and I get bug reports. As a framework developer, I would really like to use a haxe.http async load by supporting 2 simple functions.

If you have more complicated needs, you can side-step the whole thing. But I guess you would need to think about how you will let a developer call haxe.Timer to run something later on the main thread, as they would expect to with all the targets with a multi-platform toolkit.

[waneck]

After giving some thought about it, and talking to Nicolas and seeing Hugh's comments about it, I understand that the use case I have in mind (mostly asynchronous IO) is not the point of this, and that it won't help there. I'd ask though, if we could please include a non-static API so that we could have other kinds of Loops that are not "THE" main loop, and allow standard library APIs (e.g. Timer and Http) to optionally select which loop they will run on. I think this change will allow this to not fuddle with the async IO use case. Otherwise, any async IO implementation will have to override Http and other types anyway, which in my humble opinion defeat the purpose of this.

[hughsando]

Maybe another way of thinking of it is that its api is really just "runOnMainThread(f)" plus "runTimerOnMainThread(t,f)" as far as a non-framework developer is concerned. So you could have http.requestAsync(url, onComplete), and onComplete would allow function() MainLoop.runOnMainThread( function() doSuff() ) - No explicit mention of MainLoop in the async api. Or maybe http.requestAsync(url, onComplete, ?where:(Void->Void)->Void) Here, "where" could default to "MainLoop.runOnMainThread", you could provide "myLoop.run" if you wanted, or just function(f) f() which could maybe be "MainLoop.runImmediately".

I guess I am saying that since the api is really only 1 function, closures can work as well as or better than instances.

As for timer, I think it is a bit different. The key function is delay (f:Void ‑> Void, time_ms:Int):Timer, which is static. Rather than providing a "where" here you can add the function to your loop object: myLoop.delay (f:Void ‑> Void, time_ms:Int):Timer. You can of course do this today. I say this mainly because the implementation of adding a "where" here is not immediately obvious. If Timer.delay were made into a "dynamic" function, frameworks would be able to override this very easily, although I do not see the use case here, since overriding the main loop gets you a little mode flexibility.

In summary - MainLoop is only for the main thread. Async apis should allow callbacks on things other than the main loop, but make it easy for it to be on the main loop.

[ousado]

My concern is this:

• people will start writing code using this API
• people wish to have their code, some async IO or whatever, run on one or more parallel event loops (and that's not an uncommon or rare use-case at all)
• people will notice that's impossible with haxe.MainLoop
• people will realize that they will have to rewrite their code
• if haxe.MainLoop is the only event loop available on their platform at the time, they then will realize they'll even have to write the event loop implementation - possibly by adopting MainLoop just enough so it's not a global thing anymore..

That's such a bad idea, so absolutely unnecessary and so easy to avoid, I really don't know where to start.

[bjitivo]

Can someone explain to me what it even means to run code in "multiple parallel event loops" in a single threaded program? A single threaded program can one exactly one stream of code at a time. That stream of code is managed by a single event loop in the most basic proposal for event loops. If you have multiple threads (i.e. the ability for the runtime to run more than one stream of code concurrently), then each of those can run their own event loop if desired. What would it mean for a single thread of execution to have "multiple main loops"? When would each one run? And what would be the advantage over collapsing everything into one main loop since that thread can only one run function at a time anyway?

[bjitivo]

I guess I'm asking for an example to back up this claim:

"people wish to have their code, some async IO or whatever, run on one or more parallel event loops (and that's not an uncommon or rare use-case at all)"

In my experience, event driven programs do not manage multiple main loops. I would love to see an example program where more than one main loop made sense in a way that wasn't just as easily accomplished with a single main loop.

[jdonaldson]

I think it's also worth considering error catching behavior. Async and a while(true) loop are going to behave differently.

[montibbalt]

+1 on keeping it as a haxelib for several reasons:

• I don't want my developers to use this even accidentally and relying on DCE for that is pretty weak
• It seems extremely silly to add something like this directly in haxe right now when the RC milestone is already past due, we're not even going to attempt to do it right the first time (people in here are still arguing on what "doing it right" even means), and a haxelib gives you infinitely more flexibility on developing/releasing/updating it

[ousado]

@bjitivo

If you have multiple threads (i.e. the ability for the runtime to run more than one stream of code concurrently), then each of those can run their own event loop if desired.

Exactly - they should be able to do that - but with the currently proposed implementation they can't, because there's one global MainLoop only, which is exactly my point.

I guess I'm asking for an example to back up this claim:

"people wish to have their code, some async IO or whatever, run on one or more parallel event loops (and that's not an uncommon or rare use-case at all)"

Glad you're asking - I think I have a few examples handy,

• zimg.com - "A lightweight and high performance image storage and processing system.", open source, so here's exactly where each thread in the http processor is initialized with its own event loop: https://github.com/buaazp/zimg/blob/master/src/main.c#L386
• RProxy: "RProxy is a reverse proxy server written with performance and scale in mind." - also open source (just not updated on github since the author and maintainer of the open source version left the company, as far as I'm aware) https://github.com/mandiant/RProxy/blob/master/src/rproxy.c#L1454
• Erlang every ERS that runs on a multi-core CPU and is configured accordingly binds schedulers to OS threads in a configurable way - which essentially is the one event-loop per thread model with some sugar on top: http://erlang.org/doc/man/erl.html#+sbt
• Tora, an example from Haxe-land, also uses one event loop per thread: https://github.com/HaxeFoundation/tora/blob/master/src/Tora.hx#L904

And there are many more - it's not like it doesn't make sense to have several threads with independent event loops that essentially do the same work and do (or even don't, but that's more debatable) require some shared, perhaps immutable state, and are demanding CPU-wise, like e.g. the SSL/TLS handling in RProxy.

[bjitivo]

I think it's reasonable for the compiler to emit a preamble that runs before the main() function that initializes an event handling loop for the thread that is about to run main(), which can thus be called the "main loop", and then continue to run the main loop after the main() function as long as there continues to be callbacks scheduled on the main loop.

I think it's also reasonable for any thread to be able to create an event loop and run it if desired.

Guaranteeing that there is a thing called the "main loop" that can have work scheduled on it allows a lowest common denominator of event loop functionality, where all haxe libs and applications can at least assume that there is one event loop onto which work can be scheduled; libraries can advertise that they always schedule work on that loop, or if they want to be more sophisticated in cases where threading performance is paramount, can also take the event loop onto which to schedule work as an input parameter, allowing the application to select which threads will do which work.

I would expect 99% of applications to function perfectly well without ever having to create another event loop; those specialized programs which (typically without justification beyond premature optimization :) really require scheduling work onto multiple event loops will still be able to do so.

[ousado]

Oh, and I forgot, another reason to want to do that, at least according to quite many questions on various mailing lists, especially for inexperienced users, is when they realize they have some longer-blocking-than-expected operations somewhere their main loop, and want to move them out of the way without much hassle.

[bjitivo]

They'll only get to do that on platforms with multiple threads. And turning users not sophisticated enough to understand how to break long running work up into pieces so as not to clog up the main loop loose on threads is just a very, very bad idea in my experience :)

[ousado]

yes, I too tend to encourage them to rewrite these dreaded database drivers that all too often only come in synchronous versions into non-blocking code, but somehow they rarely listen (or never show up again) :)

[ncannasse]

Having several loops does make only sense in a multithreaded application, which requires very custom synchronization primitives, it's entirely outside of the scope of this library to provide an abstraction over this. Given that it was the main comment since my last call-for-feedback, I've merged my proposal.

[markknol]

Would it be nice if MainLoop.add would accept f:Float->Void as argument, and pass the delta time? For game dev this is very common.

MainLoop.add(function(deltaTime:Float) {

});