google / marl

A hybrid thread / fiber task scheduler written in C++ 11
Apache License 2.0
1.89k stars 193 forks source link

Launch tasks in the thread init function #267

Closed peroket closed 3 months ago

peroket commented 3 months ago

Hello,

I am using this library, and it works extremely well, thank you. My current setup is to launch a bunch of threads and each thread is running its own independent tasks, so I just call Scheduler::bind() and it's all working well.

On top of that, I have tasks that run forever in separate threads without marl. These are also event based, and I want to convert them to fibers to not have extra threads, but I don't want to burden any particular scheduler threads with additional tasks (they don't need thread locality).

My idea is then to launch a scheduler with the number of threads I need, use the init function to launch the tasks that should remain thread local (and make sure I have them running in all launched threads), and then launch the tasks that can be run on any thread via the scheduler, which do its thing and then create a new task to process the next event on any thread available and share the load.

However, the init function does not allow that at the moment, because as far as I can see I do not have access to the worker, and the thread local variables are set only after it has been called (https://github.com/google/marl/blob/main/src/scheduler.cpp#L377). I would like to know, then, is there another solution? Would it be OK to move the binding of the thread local variables to be able to call Worker::getCurrent()->enqueue() from within the init function?

peroket commented 3 months ago

Note: I did the change here, and it seems to work fine https://github.com/google/marl/compare/main...smlxl:marl:main

peroket commented 3 months ago

This turned out to make my project much slower, so will not do it that way.