Std Main Loop

[ncannasse]

Each platform comes with its own “main event loop” implementation, making it hard to build crossplatform API over it.

We shoul provide a base abstract loop definition that can help this.

Discussion opened on the topic.

[Simn]

I have to wonder if this really practical. Maybe @hughsando can share his thoughts on this matter.

[delahee]

I dunno if it's realistic but it is one of my wish. Just like in many other languages ( which are less constrained off course) start the main func have a gl renderer and make games :)

I was even able to do it in prolog and caml why not haxe :) Le 29 mai 2014 17:29, "Simon Krajewski" notifications@github.com a écrit :

I have to wonder if this really practical. Maybe @hughsandohttps://github.com/hughsandocan share his thoughts on this matter.

— Reply to this email directly or view it on GitHubhttps://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-44544929 .

[Justinfront]

On 29/05/2014 17:41, delahee wrote:

I dunno if it's realistic but it is one of my wish. Just like in many other languages ( which are less constrained off course) start the main func have a gl renderer and make games :)

You can get GL context on several targets so if your happy to fight with GL coding I think you can do exactly what you suggest currently eg:

http://old.haxe.org/doc/java/lwjgl

var gl = Browser.document.createCanvasElement().getContextWebGL( cast {alpha:false, depth: false, failIfMajorPerformanceCaveat: true });

http://chronosyndrome.com/?p=30

I wondered about Python maybe the best approach is to think about how Haxe and it's targets could integrate with swig.

http://www.swig.org/tutorial.htm

I tried to use -net-lib on mono with Cairo but not yet working.

But still keen on a Sys style approach to GL.

[hughsando]

I'm not even sure what this would look like. Maybe for js and flash, the main loop looks like this: // Flash implementation: // JS:

And nme can simulate the (non) api on its targets.

There are a couple of ogl implementations around. NME uses the webgl api, and there are some other efforts going on that attack this from a different direction. For example, nme wraps the integer id so you do not leak resources. But all this relatively simple and is not really the problem. The problem comes with creating packages (mixing java an ojcc), asset conventions, touch events, audio, game center plugins, in app purchases, fonts etc etc. For example, I notice h3d has its own asset system - but does this cover audio? If so, then it is more than "3d", if not then the user will need something else as well. I don't think we can talk generally about some standard loop, I think it will be that libraries are made compatible. Eg, "h3d or flixel" or "h3d for nme" are easy to integrate and mix-and-match, but this does not require a standard.

[ousado]

How does an abstract "Main Event Loop" imply a GL context all of a sudden? An abstract event loop should be just that, and not imply any heavy dependencies like graphics libraries.

[delahee]

Sorry for the digression. The main point the foundation evoked stays true. Entry points are fragmented which means platform specific code to have starting main loop :) Le 31 mai 2014 19:27, "ousado" notifications@github.com a écrit :

How does an abstract "Main Event Loop" imply a GL context all of a sudden? An abstract event loop should be just that, and not imply any heavy dependencies like graphics libraries.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-44754269 .

[elliott5]

I'm assuming that before the "Main Event Loop" starts, there is no concurrency. During this phase, while all of the set-up and initialization of state occurs, concurrency is undesirable. This is because if there were concurrency during the set-up phase, the results could be different for each run of the program.

Once the program enters the "Main Event Loop" phase, concurrency can occur. I'm assuming that the "Main Event Loop" would provide a platform-independent way to schedule call-backs, based on system or user-created events. Please tell me if I am wrong.

But how do threads fit in? And more importantly, how can the thread paradigm be made to work cross-platform, so that threads can become a standard tool for writing cross-platform concurrent Haxe?

As this is exactly the problem I am faced with inside TARDIS Go, in order to emulate goroutines, it is much on my mind. I emulate goroutines at the moment by generating Haxe code that is the subject of repeated call-backs, with each goroutine remembering its stack and program-counter state in between each call-back.

A similar approach may be possible for general Haxe code wishing to run as a thread when the target environment does not support threading. Here the envisaged "Main Event Loop" could be really important, as it could provide the stream of call-backs required to execute the emulated thread.

Why emulate threading? Because it produces code that is easier to understand and reason about than call-backs do.

I believe that others in the Haxe community, with much more knowledge than I have, would like to see continuation-passing style ( http://en.wikipedia.org/wiki/Continuation-passing_style ) used as the implementation mechanism for these emulated threads.

As I'm new to Haxe, there is every chance that I have said something very stupid above; so please correct me.

[georgkoester]

I am working on integrating Haxe code in a program that has its own main event loop. I would just like to have that remain possible - nme makes this really hard and I wouldn't want Haxe to go into the same direction.

[ncannasse]

Ok, I'm starting to push some ideas around this. Here's a first concept of haxe abstract "MainLoop":

https://gist.github.com/ncannasse/8674575efe456cdca5ed

It's very basic, just a list of pending events with priority. My idea is to have the Haxe compiler automatically insert a MainLoop.run() (if MainLoop.hx is referenced and not DCE) after the Haxe main().

Systems that want to redefine the MainLoop would only have to respect the API so users can insert their own events in a crossplatform manner.

This could be slightly improved to add a delay parameter to events so we can do proper scheduling without having to run "infinite" loops.

Would like to get @hughsando @underscorediscovery @jgranick @RobDangerous feedback on this, so we can generalize it for all windowing systems (nme, lime, snow, kha)

With this we could even implement Socket events and Timers on platforms that don't have an actual loop such as Neko.

[ruby0x1]

My gut feeling is that it is difficult to make sense of without being heavy handed or invasive.

In your given example: having no control over what goes inside while(true) is just not an option for many use cases. I wonder too about callbacks on C++ being Dynamic calls and not being optimal yet - this is overhead on every tick now we're forced upon.

If the user would call MainLoop.tick() and not run() once, this would work already because platform differences. There are many other issues like exclusivity and control.

It's important to consider platforms like iOS and Web run the main loop for rendering applications entirely differently, there is no while true and there can't really be a loop like this. Mobile has heavily platform specific ways that tie into the application life cycle and removing control over these aspects for the end user to facilitate this is quite impractical and for me not an option. A Haxe application (if you choose to build your code as one) is just an entry point and not an application which I think is an important separation – the entry points, loops, and more are dictated by the platform and not the programming language - otherwise you are no longer targeting languages but runtimes.

For things (like Elliot mentions) such as Coroutines (of which there is an implementation which just requires a scheduler/event loop), Promises, and other language level constructs that would be great to add to Haxe - and things that require updates like Timers and Sockets - The concept would be a welcome/required addition - however I think it needs care to not be too heavy handed or invasive if it even is possible to straddle both of these.

My feeling is that the focus and attention should be on facilitating language constructs and services like coroutines, sockets, timers - where the implementation can be handled in a target specific way rather than trying to dictate the application life cycle for a bunch of targets, platforms, and use cases that Haxe itself as a programming language has no concern with.

[ncannasse]

@underscorediscovery thanks for the feedback.

Regarding "every tick" overweight : we're talking here about a few closure calls per "frame", I don't think that's an overweight. The goal is not for users to register hundreds of events that get called every frame. If they have too many timers for instance, we could plug a single "timers handle" event that would then process all the timers.

Regarding iOS or exotic runtime loops : I understand that sometimes things can be different, but this simple MainLoop is abstract enough to be implemented differently whatever the platform. If you want for instance to just register an Android activity and return immediately, it's entirely possible. The only guarantee you have to give is to call "often" the events that the user has registered.

Regarding if this should be a common construct for cross platform: we have been hearing quite often from our users that it is difficult to extend existing frameworks with reusable code. I think that it's the role of Haxe to provide such abstractions when possible on which frameworks can be built on. Again we are not imposing a particular implementation, only the API/specification.

[delahee]

Dynamic access is not much a problem if it doesn't generate alloc and the code you shown seems devoid of it, so fine the overhead is nil.

As to the overall problem, I think it' is a very good move to propose a main loop specification. This will help the ecosystem to have entry point proliferation and that is a very good thing.

I do hope we'll see more "main loop" libraries and even a "shallow hxcpp/neko" loop and stuff like that that will ease user to create their frameworks upon basic loops rather than embed huge frameworks and use 5% of it.

So...I don't have true feedbacks except, go on this is a good thing to me.

2016-03-05 16:38 GMT+01:00 Nicolas Cannasse notifications@github.com:

@underscorediscovery https://github.com/underscorediscovery thanks for the feedback.

Regarding "every tick" overweight : we're talking here about a few closure calls per "frame", I don't think that's an overweight. The goal is not for users to register hundreds of events that get called every frame. If they have too many timers for instance, we could plug a single "timers handle" event that would then process all the timers.

Regarding iOS or exotic runtime loops : I understand that sometimes things can be different, but this simple MainLoop is abstract enough to be implemented differently whatever the platform. If you want for instance to just register an Android activity and return immediately, it's entirely possible. The only guarantee you have to give is to call "often" the events that the user has registered.

Regarding if this should be a common construct for cross platform: we have been hearing quite often from our users that it is difficult to extend existing frameworks with reusable code. I think that it's the role of Haxe to provide such abstractions when possible on which frameworks can be built on. Again we are not imposing a particular implementation, only the API/specification.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-192674279 .

David Elahee

[ruby0x1]

I don't understand how "often" can translate to "main loop in every framework" then. I also don't understand the usage pattern from the user (framework) perspective either because the code shown is not compatible - so if for instance on js will you run setTimeout(...) instead of while(true) – This can't be used to run a rendering main loop. So what is the framework supposed to use it for?

As mentioned, I think the idea of an event loop is a good one, but the idea of trying to abstract this away from the user doesn't align as described. If the construct is supposed to be initiated by the user (and not implicitly by Haxe) then it doesn't answer important questions like sockets and coroutines and promises needing an implicit event loop - which makes me curious if the solution is solving a problem that is being constructed to fit multiple situations that don't overlap.

The supposed guarantee just doesn't match requirements for my mind. I also don't consider a few conditions and null checks light weight for the % of use case described in the very hottest path code. If I measure by % of the hot path vs % of described uses - you say only a few - then I can't not ask why 100% of the hot path code is affected with this overhead that wasn't there before.

we have been hearing quite often from our users that it is difficult to extend existing frameworks with reusable code.

If this is referring to mobile specific "extensions" this extends much much further than simply having a function called "often" and would not be solved by this directly.

[nadako]

Wouldn't it be sufficient to extract the body inside that while(true) in a separate tick function and call it in different ways depending on platform (e.g. setTimeout/setImmediate on JS, while(true) on some custom platform, a os-specific callback on mobiles, etc)?

[RobDangerous]

I don't see any performance problems here, it's not a hot path just because it's in the main loop. I agree with @delahee - as long as it doesn't force allocations on us it's fine. Only problem I see: "often" is not a very precise specification, I would propose to at least add per frame callbacks. Kha, for comparison, just provides one callback that's called per frame and can optionally run a complete Scheduler on top of that.

PS: Here are the proper instructions for main loops on iOS: http://kode.tech/snippet-1-own-the-main-loop-in-ios/

[ncannasse]

@underscorediscovery In JS case I guess the run would simply do something such as:

static function run() {
#if js
    requestAnimationFrame(function() {
      tick(); // process events
      run();
   });
#else
   while( true ) {
      ...
   }
#end 
}

Coroutines, promises, sockets etc : every high level async feature needs to boil down somehow to an event loop in the end, they can be built on top of this. We don't have any actual way to implement cross platform asynchronous things in Haxe and that's what I'm proposing to solve here. We can think afterwards at how we can implement higher level things.

@RobDangerous yes I agree with the one-time-per-frame spec in case there is some display occurring.

[back2dos]

I cannot say much about this as I am yet to find a polite way to say what I think of both the design and the implementation of the current proposal. Maybe if MainLoop.add(function () {}); MainLoop.run(); didn't enter an infinite loop (caution: running the sample will make your browser hang) I'd find it easier to verbalize my actual concerns. I don't mean to be harsh, I just find it genuinely alarming that everyone seems to have formed an opinion without even running the code. That is no way to arrive at a practical solution, let alone at one that would be worth becoming a standard.

As far as autowrapping is concerned, you can accomplish that with macros. It requires addGlobalMetaData to register a build macro which skips all classes but the -main class which can be determined by parsing Sys.args().

There is no reason to integrate this with the compiler, so I propose to develop it as a haxelib and discuss standardisation once the code was actually used in the field under a wide spectrum of use cases.

[RobDangerous]

Hey @back2dos the way I understand it this discussion is not about the implementation because that would eventually be provided/overridden by the framework depending on the needs of the target system.

[ruby0x1]

@ncannasse you asked specifically about generalizing it for windowing.

[ruby0x1]

@ncannasse to further clarify, I don't actually see what it is that is being suggested because the topics have varied now and I feel I missed the original intent.

Can you please clarify the idea, such as what the framework might do (extend and override? call .run() only? call once per frame instead? nothing?) and what the std lib aims to provide?

In my current understanding, the only goal is an API that would allow end users, libraries, or language features to rely on: i.e by calling MainLoop.add they will be know that it will be updated by something for them, and will be guaranteed such by the MainLoop construct so they can progress over time.

[back2dos]

@RobDangerous Then I say this is an ivory tower debate around some abstract whatsoever that does not even have a clearly defined goal and I highly doubt that anything good will come out of it. Which leads me back to my original stance: put it in a haxelib.

If this is really about what @underscorediscovery said, then this solves it:

package haxe;

interface Scheduled {
  function cancel():Bool;
}
interface Scheduler {
  function add(task:Void->Void):Scheduled;
}

Every framework can provide implementations at will. Ones that are locked down with FPS or ones that are tied into the actual native loop or whatever. Slamming this into some static class that frameworks need to shadow will simply cause conflicts and thus incompatibilities. It also takes away control from the user.

Code that can leverage a scheduler can be written against an interface such as the above. Users can still decide to then run it on an eager scheduler if they want the result synchronously or what not. Also, if this is meant to be minimal, then prioritization is out of place. That should be built on top because it's not something you want to have replicated in every single implementation.

[hughsando]

I can think of only a couple of reasons for some kind of standard main loop. In console apps when the developer starts a timer or thread an expects the program to hand around while these are pending. It is not clear weather the console loop should wait for threads to finish - I think it will depend on application. To this I might add async io, if you want a generic async haxe.http. So I propose entering a blocking loop at the end of main, unless a "asyncWake" callback has been registered (see later), while there are timers pending. To be cross platform, the timer callback should get run on the main thread. For browser + flash the loop is only notional, like it is now.

You could also consider adding a sys.Thread class which can take an extra parameter to the create function, "waitForTermination", or maybe "Scheduler.createThread(callback)", with this set, the sys.Thread class wraps the user function with another function that increases/decreases a "current waiting thread count" and pokes the even loop when the count drops to zero and otherwise prevents the main loop from exiting.

Any kind of "onFrame" callback, or even a "tick" presupposes a concept of frames or at least keeping the cpu awake, and should not be part of the standard.

For async io, eg http onload/onerror, the main loop would also block on console apps while there are pending requests. The easiest way might be to simple spin up a thread-per-request, and then use the thread-exit mechanism described above. libCurl allows you to combine all sockets into a single thread - haxe http could do this too, or use a poll. For browser, this is handled natively already. For flash, you would create background loaders - no need for main loop. For node, I guess they have a pretty good solution here. For gui frameworks, I'm not too sure. Nme solution here is like the timer solution, that is "poll" and "get next poll time". With no timers and no callback pending,the poll time is infinite. Other events (such as mouse clicks) cause the next poll time to be recalculated, and other events (eg, threads) can generate asyncWakes. async io and timer callback happen in the "poll" callback to ensure the thread is known and consistent across targets. How the async io handlers register themselves I do not have much of an opinion about, but nme uses a "poll Client" https://github.com/haxenme/nme/blob/master/nme/app/IPollClient.hx But it should be pretty simple to adapt. This will need to be platform specific, because this does not make any sense for browser or flash. I guess the haxe.http class will look completely different in these cases.

So I propose a solution, say, haxe.Scheduler, with "timeToNextPoll" and "Poll", and "asyncWakeCallback" for the frameworks. Adding a timer, async io or a thread finishing cause the nextPollTime to change, and an generates a asyncWake callback. For targets with built-in loops, these functions simply do not exist. For console apps, the main loop is blocked while there are pending events. For frameworks that want to control the main loop, they must respect the "timeToNextPoll", if it exists and call "poll". They also need to register a "asyncWake" callback, that allows newly scheduled timers and async io to get serviced when there are no other callback pending. Developers wishing to schedule a poll or keep-alive for their own async io will need to be aware of the target, but can implement the pollClient on framework/console targets.

This should give timers and async io on all platforms and thread-waiting for console apps. I think adding thread-waiting to frameworks is not great, but could simply be achieved via a Scheduler function "areThreadsRunning()", and cancelling the window-close if you really want.

[ncannasse]

@back2dos I think you need to read a bit more before writing :) The example code you are showing is exactly supposed to enter an infinite loop, at least on a "system" platform such as neko or php. You're registered a new event loop and it will be run until stopped. Of course on JS/NME/Lime/Snow this would not freeze but instead refresh the window then call the event depending on the screen refresh rate. Read more about why it's useful in Hugh comment.

@hughsando yes I think we might want to have poll'able events as well, but can't this be built on top of what I'm proposing? For instance if you use epoll() for managing your sockets, you would epoll() every tick(). You can eventually epoll() infinitely if there's only one MainEvent registered (which would be the case if you're only waiting for a Http call without any display occurring). Or you could have threads that do that in the background, then add a MainEvent in order to make sure the callback is run into the application thread.

Please clarify your haxe.Scheduler API proposal as a GIST, we should not assume that we can have multiple threads running on the platform.

@underscorediscovery I would like this to become a standard that Haxe users can rely on, so for example one that uses Snowkit would be able to use this API and get things called as specified (the current discussion is about discussing the specification), so yes in order to work correctly Snowkit will have to support it since using the default implementation would freeze the application as Juraj cleverly noticed.

[ncannasse]

BTW, I really want that we collaborate at finding a solution that is acceptable (if not perfect) for everybody, so if you have issues with the API proposal I made, please think about it and propose your own, or else you will be bothered in a few months when users start using it and ask you to add support for it in your framework :)

[back2dos]

@ncannasse Thanks, I think I've read enough. I see my feedback is not welcome. Have fun.

[ncannasse]

@back2dos don't take it badly, your feedback is welcome as long as you're trying to understand what we're trying to do, and is being constructive.

[hughsando]

Juraj, I think you have been distracted by "MainLoop" which should be considered only one possible solution. Instead, perhaps you could take a step back offer an opinion on the problem: How can the haxe std libs, particularly http, socket io, database io, File.getContent, directory.read, as well as custom libraries, such as "on filesystem changed" interact with a command line main loop, a browser main loop or a framework main loop (nme/opengl/kha/heaps/waxe/...) in a consistent way.

I'm thinking browser (js/flash) will be significantly different to the other targets, but I could be wrong.

On Sun, Mar 6, 2016 at 5:08 PM, Nicolas Cannasse notifications@github.com wrote:

@back2dos https://github.com/back2dos don't take it badly, your feedback is welcome as long as you're trying to understand what we're trying to do, and is being constructive.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-192842533 .

[ncannasse]

@hughsando the only difference I see with browser (and maybe mobile as well?) is that we need to actually return from our haxe main() after registering ourselves into the plaform loop that we don't own, whereas on other platforms we need to keep running until we're "finished" (which my MainLoop defines as all events have been stopped, but that could be all windows have been closed, all threads have terminated, etc.)

[hughsando]

Nme registers an "onWindowCreated" callback and exits - openfl used to, not sure anymore. Waxe exits too. Flash/js obviously so. Command-line is the only "framework" which needs to block after the main AFAIK.

Either way, enforcing that a framework sit in THE loop, makes is completely unusable for me (nme,waxe).

On Sun, Mar 6, 2016 at 5:47 PM, Nicolas Cannasse notifications@github.com wrote:

@hughsando https://github.com/hughsando the only difference I see with browser (and maybe mobile as well?) is that we need to actually return from our haxe main() after registering ourselves into the plaform loop that we don't own, whereas on other platforms we need to keep running until we're "finished" (which my MainLoop defines as all events have been stopped, but that could be all windows have been closed, all threads have terminated, etc.)

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-192854301 .

[back2dos]

@ncannasse I am sorry for not having understood your code. It's just that virtually all major runloop implementations out in the field will execute scheduled callbacks once. This is true for node's process.nextTick, for the DOM's window.requestAnimationFrame, for Cocoa's NSObject:performSelectorOnMainThread, Swings SwingUtils.invokeLater and what not. I will leave it open to speculation who's lack of reading is the cause of my inability to understand your code. Let me leave it at reemphasizing that with this particular choice, it will be close to impossible to integrate decently with most environments, which is a point that others have raised. Please do take that seriously.

That is not my point however. My point is: this - whatever "this" is - is a horrible idea. This thread is filled with implicit assumptions none of which hold. The result will cause more harm than good. I do not have the time or energy to discuss that on some theoretical level. Problems will occur naturally once you try to integrate this supposedly abstract API with other systems. My proposal was thus to put it in a haxelib to contain damage (and also - instead of adding to @Simn's frustration over pending issues - have a separate tracker to deal with all the problems that I predict you will have with this) and elevate it to a standard once it is actually shown to be usable. I have also pointed out an alternative to baking this into the compiler. Both proposals are not only constructive, but even directly actionable. And I would argue that they are pragmatic too, leading to tangible results more quickly.

@hughsando As for your concrete question, I think it is a very interesting one, but it is remotely related at best. Tautological as it may seem, the short answer is: for asynchronous IO you need abstractions for asynchronous IO - and not abstractions for run loops. Interfaces that act as the asynchronous counterparts to haxe.io.Input and haxe.io.Output. Some implementation may indeed be backed by a synchronous input/output and a way to offload IO to a background thread and have the result be dispatched back onto a main loop of sorts. Other implementations may simply rely on what the native runtime environment provides. I have published work in this area a few months ago that provides a homogeneous TCP and HTTP API for nodejs, java and neko. We can gladly discuss what little I was able to learn and the countless things still left to do, but I don't think this particular issue is the right platform for that.

[hughsando]

I have put an outline here: https://gist.github.com/hughsando/91a19a61a82bf8b7a9e6

The basic idea is that frameworks can call "poll" and "getNextWake" directly and the console can block and do it automatically.

If you do not wish to add the thread stuff, we can just block if there are timers pending - most frameworks already support the haxe.Timer api, although this does not allow for a Zero-CPU solution.

As for haxelib vs std lib, incubating in a haxelib seems like a good idea, but if you convert haxe.Timer over to it, maybe not.

On Sun, Mar 6, 2016 at 7:12 PM, Juraj Kirchheim notifications@github.com wrote:

@ncannasse https://github.com/ncannasse I am sorry for not having understood your code. It's just that virtually all major runloop implementations out in the field will execute scheduled callbacks once. This is true for node's process.nextTick, for the DOM's window.requestAnimationFrame, for Cocoa's NSObject:performSelectorOnMainThread, Swings SwingUtils.invokeLater and what not. I will leave it open to speculation who's lack of reading is the cause of my inability to understand your code. Let me leave it at reemphasizing that with this particular choice, it will be close to impossible to integrate decently with most environments, which is a point that others have raised. Please do take that seriously.

That is not my point however. My point is: this - whatever "this" is - is a horrible idea. This thread is filled with implicit assumptions none of which hold. The result will cause more harm than good. I do not have the time or energy to discuss that on some theoretical level. Problems will occur naturally once you try to integrate this supposedly abstract API with other systems. My proposal was thus to put it in a haxelib to contain damage (and also - instead of adding to @Simn https://github.com/Simn's frustration over pending issues - have a separate tracker to deal with all the problems that I predict you will have with this) and elevate it to a standard once it is actually shown to be usable. I have also pointed out an alternative to baking this into the compiler. Both proposals are not only constructive, but even directly actionable. And I would argue that they are pragmatic too, leading to tangible results more quickly.

@hughsando https://github.com/hughsando As for your concrete question, I think it is a very interesting one, but it is remotely related at best. Tautological as it may seem, the short answer is: for asynchronous IO you need abstractions for asynchronous IO - and not abstractions for run loops. Interfaces that act as the asynchronous counterparts to haxe.io.Input and haxe.io.Output. Some implementation may indeed be backed by a synchronous input/output and a way to offload IO to a background thread and have the result be dispatched back onto a main loop of sorts. Other implementations may simply rely on what the native runtime environment provides. I have published work in this area https://groups.google.com/forum/#!msg/haxelang/2gRl_zCc2-0/VHUUwTp6AgAJ a few months ago that provides a homogeneous TCP and HTTP API for nodejs, java and neko. We can gladly discuss what little I was able to le arn and the countless things still left to do, but I don't think this particular issue is the right platform for that.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-192875814 .

[back2dos]

On Sun, Mar 6, 2016 at 1:34 PM, Hugh Sanderson notifications@github.com wrote:

As for haxelib vs std lib, incubating in a haxelib seems like a good idea, but if you convert haxe.Timer over to it, maybe not.

During that period such a haxe.Timer implementation could simply be part of said haxelib.

[Simn]

I think that

• we (the Haxe Foundation) should manage it so that we end up with a centralized solution that works for everyone, and
• it should be a haxelib so that I don't feel like I have to maintain it we can make independent updates.

[ruby0x1]

• I think the haxelib idea is better
• I agree with Hugh and Juraj and have stated: as described it is not something I feel will solve the right problems.

It is overly invasive for me to feel comfortable with - at least as described. Ceding control over something as fundamental and crucial to cross platform portability of the system level loop cycle is just not an option for me, nor nme or others which has been stated as well.

The notion is good but the suggested implementation and ideals (of replacing every main loop) won't just doesn't fit this way.

(p.s snowkit is the community name, snow or luxe are frameworks)

[dazKind]

IMO, put this into a simple haxelib. if it proves to be a viable thing that people need and use you can easily merge it into std.

[ncannasse]

@back2dos saying at the same time this is a horrible idea AND that you don't have enough time to explain why, without giving even the slice idea how why you would think that, is not very nice to say the least, and definitely not constructive. Not that I want to enter a heated debate on this topic, please simply make a contribution by proposing an alternative that can be discussed.

Choosing to go haxelib or haxe std will depend on how complex the code is and how likely it is to be changed in the near future. This will only happen after we have decided on it. I didn't happen this way but if everybody had agreed with my first proposal I don't see what value it would have to put such simple piece of code on haxelib.

We want a solid piece of foundation code that is shared across frameworks and that people can build over.

@underscorediscovery the idea is not to cede the control, but to let users add their own systems to it, with a minimal specification that the framework will be requested to follow. The framework will of course have its own implementation that is optimized for its own subsystem. I don't see how that can be something bad or that doesn't solve problems.

[ncannasse]

@hughsando thanks for submitting your idea. I find it a bit too much oriented towards threaded/native systems, I'll try to improve over it.

Regarding your clients management: using a double linked list as I did in my MainLoop ensure that removing clients that are currently / will be run is correctly supported.

[ncannasse]

@underscorediscovery could you also propose something that would work for you?

[hughsando]

I don't imagine adding/removing will happen very often.

But as far as threads go, you can assume these are just "#if" ed out and you can ignore them on non-native targets. BUt the whole thing practically disappears on browser targets, so what problem are you trying to solve?

I do think you have to run the callbacks on the main thread or else you will complicate the developers code.

On Sun, Mar 6, 2016 at 11:42 PM, Nicolas Cannasse notifications@github.com wrote:

@hughsando https://github.com/hughsando thanks for submitting your idea. I find it a bit too much oriented towards threaded/native systems, I'll try to improve over it.

Regarding your clients management: using a double linked list as I did in my MainLoop ensure that removing clients that are currently / will be run is correctly supported.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3075#issuecomment-192917257 .

[ncannasse]

Here's a second version that allows to delay() an event, and which allow another thread to wakeup() the main loop.

https://gist.github.com/ncannasse/8048e759daa6cea3ce8c

There's still a good number of things to do wrt threads safety, such as protecting the events list, we can add these afterwards, nothing hard there. The code is just to give you an idea.

@hughsando yes, you want of course have the callbacks run in the main thread. For this in my MainLoop you simply add an event with no delay that will be run once and stopped.

@back2dos the reason why events are run in loop by default instead of only once is to reduce allocations. For one-time event you stop() when its called first. For many-times events (setInterval like) you have nothing to do.

As an integration for instance in Snow/Luxe @underscorediscovery, that would mean that you have to include haxe/MainLoop.hx in your framework, redefine run() only, optimize a few things that might be relevant, and somehow make sure that tick() is being called

The problem I'm trying to solve is to have a crossplatform way to develop async code. ATM we lack the constructs and it requires #if'ing for each framework available, which is not a good. I want this to be:

a) lightweight b) efficient c) adopted by all frameworks

[delahee]

I think Hugh has got a nice lead here. If we could simplify a bit, it would be nice. Nicolas (first) approach is interesing too but I kind of worry it is too straightforward in this Activity/Delegate/requestAnimationTimer hostile world.

As to haxelib or not, despite what I heart, my brain says the std.mainloop could go in an official haxelib just like spod & dispatch should.

Annnnnnd since there are mostly frameworks writers here ( which are sparse users of third party libs ) and since tivo was very vocal about main loop, let's bug them :)

ping @kulick

[Simn]

You're probably looking for @bjitivo.

[ruby0x1]

I have earlier,

If the user would call MainLoop.tick() and not run() once, this would work already because of platform differences.

The second one with the tick fits closer to what makes sense in practice. Who is supposed to call run? Is it called implicitly at the end of main()? If so, I will override run with a blank function and retain the control needed. Calling tick explicitly to update the events is perfectly fine from my view - it is the least invasive and retains user control over it being there in the first place.

To me it makes sense because:

• CLI/other use case can add explicitly an event into the queue
• Std lib can add events into the queue via API calls like sockets etc
• Frameworks can progress the queue themselves from whenever/wherever it makes sense to do so

It's important then that tick be the only thing doing the work, i.e there are currently structures inside run that the user would have to replicate in their own code and would be critical to it functioning like the mutex. I would move these into the tick function so there is only one point of contact that should be "called often" and no other strange coupling.

You could condense the delay as a parameter to add(delay, fn, priority) to simplify usage if it's once off.

[waneck]

I really like Hugh's suggestion. It has the concept of main loop sleeping / awakening, and it allows timed events in a way that doesn't involve infinite looping until the event it ready :)

Coincidentally, I've been working on a scheduler for the native targets, so this is very fresh in my mind. I think that Hugh's solution covers most of it, which has to do with being able to awake it when a specific event has happened (e.g. epoll_wait has returned), being able to run events on the main thread, and allowing timed events to happen.

There are some more complex cases where we might have multiple threads running epoll for example on the same set of events/sockets. This allows very efficient use of one selector per cpu, but I can imagine it might be outside this scope.

There are also some possibilities of having multiple loops inside the same application.In my own use case, I was thinking about allowing an asynchronous class (e.g. Http) to receive an optional Scheduler argument. Otherwise, it uses the default.

I'm a little worried in this sense about how each implementation will setup its Scheduler. It this based on classpath? This might get tricky depending on the scenario and set of libraries you're using.

[ncannasse]

The second one with the tick fits closer to what makes sense in practice. Who is supposed to call run? Is it called implicitly at the end of main()? If so, I will override run with a blank function and retain the control needed. Calling tick explicitly to update the events is perfectly fine from my view - it is the least invasive and retains user control over it being there in the first place.

Yes that's exactly it. Actually I think we should not have run() there but in a separate class EntryPoint.hx that would simply do MainLoop.run() so you only have to redefine EntryPoint without having to sync the MainLoop code in case we make updates.

@waneck regarding multiple event loops, we could make MainLoop an instance + a static singleton, with current static methods that would use the singleton by default.

[ncannasse]

Regarding Hugh implementation, I have a few questions/comments so we can combine with my updated MainLoop. I've tried to list all the differences:

• I tend to prefer using double linked list for clients implementation, which would allow for partial locking when manipulating the list instead of fully locking it. This should allow more thread-safe iteration on the list, and cause less copies/allocations when pushing/removing from the Array. Any disagreement?
• Is getNextWake necessary or a fixed delay such as my MainEvent.nextRun good enough? I can't imagine right now an event that would change its delay dynamically. nextRun is more efficient but requires a class, not an interface (see below)
• any poll() vs tick()/call() vocabulary preferences ?
• using interface IScheduleClient VS closure (te latter feels more "Haxe" to me)
• after giving it some thoughts, threads API seems good to me. However I think we should keep it private, so the platform thread implementation is responsible for calling it.
• should our default haxe.Timer implemenation use a single Timer Manager as Hugh suggests or should all timers be registered as separate Events as I thought would be possible in my MainLoop v2 ?

[Justinfront]

I am curious about the javascript loop in Nicolas's gist https://gist.github.com/ncannasse/8048e759daa6cea3ce8c For browser js, is a css3 injected loop better in some situations? https://github.com/Justinfront/divtastic3/blob/master/src/core/CSSenterFrame.hx and if so, what would be the setup for me to override the conventional inbuilt approach, is this going to limit developers present and future and even past by impacting old code? For instance on as3 you assume onEnterFrame but again maybe it's not quite the same for Red Tamarin, maybe it has a better setInterval or something. On java swing if I wanted a couple of threads to handle different aspects of the app is that setup, say one for render and one for interaction events? Lastly what happens when we are mixing Haxe code with platform code will this setup get in the way in how we are forced through internal politics, or historic reasons from using Haxe smoothly, for instance loading as3 into haxe into as3 movies with different types of domains. Maybe it's all cool in all situations.

[ncannasse]

@Justinfront the idea is to be able to simply redefine the run() method, so one given framework can hold the loop and do its own stuff, still calling tick() so that crossplatform events are correctly handled. I really don't think this will cause issues with AS3 domains or other scenarios.

[ncannasse]

I have put a new GIST that separate between MainLoop (cross platform) and EntryPoint (framework-specific). I think that Hugh support for threads would go in EntryPoint. https://gist.github.com/ncannasse/3614dd6b6e7f7242b60d

[ruby0x1]

• I think your notes on run make sense. Maybe it could be like how trace is reassigned, since run is a single call and not every frame it's fine if the call isn't the most direct/efficient.
• poll/tick either vocabulary is fine to me, I also like update or process but poll is common in event loops
• The only current thought I have on the next wake model is that it doesn't try and do any frame timing and just runs as fast as it's called (not related to delayed events timing obv). The only (very much) older implementation of the nextwake that I encountered was in nme which wasn't precise at the time. Timing should be in control of the user when it's tick is run and nothing trying to be fancy.

EDIT: In the gist, curious why does js type the RAF as Dynamic explicitly? it also doesn't check for it being null, and has no fallback to setTimeout (this will depend on how far back Haxe supports browsers, I only target modern browsers so I have never checked).

Otherwise it seems fine, is the purpose for the framework/user create a project local haxe/EntryPoint.hx class which shadows the default?