Development status?

[mdsitton]

Seems like everyone contributing and using this project dropped off of it back in 2021, @RichieSams what is the status of the project since so far to me it looks like the best option for an "off the shelf" implementation of a fiber based jobs system for a game that i've been able to find so far.

[RichieSams]

Hi @mdsitton. Thanks for reaching out.

We're all still here, however, as you correctly identified, active development of FiberTaskingLib has dropped off. That said, FTL is "basically" done. All the PRs and Issues are either features we thought might be a big benefit and ended up being a wash. Or just some additional "nice to have" features (like parallel for).

So FTL can absolutely be used as-is. (And we'll be here to help if you find any bugs). That said, while a fiber-based job system is awesome, just remember that there are significant drawbacks still in today's ecosystem:

1. Debuggers have no first class support for fibers. So the callstacks look a bit weird. But more importantly, you can't easily see the callstacks of "waiting" fibers. There might be a way to "register" them with the debugger. But I haven't done much looking.
2. As mentioned in the README and Christian Gyrling's talk, you can't use Thread Local Storage. FTL has a fiber-based TLS implementation you can use instead. But this can get annoying, especially when third party libraries use TLS.

[BrodyHiggerson]

Hey @mdsitton, thought I'd weigh in as well as I work with FTL regularly (almost daily). The issues @RichieSams pointed out are not fun, particularly 1., but honestly for me the worst part is lack of proper profiler support. Name any flamegraph style profiler that you like, and I would be willing to bet a small sum that they don't/won't support Fibers - they get confused when you jump around the callstacks while inside one of their begin/end profiler macros, for example.

This profiler ended up adding some hooks for profiling Fibers, https://github.com/dfeneyrou/palanteer, but you can see that the GUI isn't the best for such things: https://github.com/dfeneyrou/palanteer/issues/39.

I don't meant to dissuade you if you're experimenting or wanting to learn. That + the amazing Naughty Dog talk were why I started using FTL and have been using it for YEARS. But the lack of profiler support is a serious, existential technical threat to my work so I'm evaluating pivoting to a more traditional task library like TBB.

Regardless, FTL is a beautiful piece of work and the way you can just appear mid-callstack like magic still blows my mind.

[mdsitton]

Hi @mdsitton. Thanks for reaching out.

We're all still here, however, as you correctly identified, active development of FiberTaskingLib has dropped off. That said, FTL is "basically" done. All the PRs and Issues are either features we thought might be a big benefit and ended up being a wash. Or just some additional "nice to have" features (like parallel for).

So FTL can absolutely be used as-is. (And we'll be here to help if you find any bugs). That said, while a fiber-based job system is awesome, just remember that there are significant drawbacks still in today's ecosystem:

1. Debuggers have no first class support for fibers. So the callstacks look a bit weird. But more importantly, you can't easily see the callstacks of "waiting" fibers. There might be a way to "register" them with the debugger. But I haven't done much looking.
2. As mentioned in the README and Christian Gyrling's talk, you can't use Thread Local Storage. FTL has a fiber-based TLS implementation you can use instead. But this can get annoying, especially when third party libraries use TLS.

Thanks for getting back to me! Infact I think I already have found a few issues that may need some updates to the underlying boost context assembly code to fix so I probably will make a pull request for that infact 😅

I mean I don't really mind the downsides too much, plus I'm not too afraid of customizing certain dependencies if required from time to time. If I can get things merged upstream or bugs fixed here for issues that pop up then I'll be fine. Also what is the status of the currently open pull requests.

Some random musings, I wonder if a job library that can have fibers enabled and disabled through a compilation flag could be useful. Because technically fibers as I understand it mainly just get you an advantage to fix some execution order issues that a typical task based system can run into, to better increase system throughput. This could allow better debugging for some cases with the downside of having less ideal execution order.

[RichieSams]

what is the status of the currently open pull requests.

https://github.com/RichieSams/FiberTaskingLib/pull/139 - Probably just needs to run CI again and then merge. Though, better would be to upgrade nonius itself (instead of changing this fork)

https://github.com/RichieSams/FiberTaskingLib/pull/111 - Not sure if it's still relevant. It was made before the AtomicCounter rework. I'd need to do some diffing and chat with @martty

https://github.com/RichieSams/FiberTaskingLib/pull/69 - We'd need to rebase and re-evaluate based on the current feedback.

https://github.com/RichieSams/FiberTaskingLib/issues/134 - I almost got it working at the end, but ran out of steam. (And there's not a huge need for it. So 🤷‍♂️ . PR's welcome! lol

https://github.com/RichieSams/FiberTaskingLib/issues/107 - Possibly still an issue? I'd have to think about it again. That said, not many code bases will change the sleep behavior mid-game.

All the other issues are just feature adds / requests.

[RichieSams]

Some random musings, I wonder if a job library that can have fibers enabled and disabled through a compilation flag could be useful. Because technically fibers as I understand it mainly just get you an advantage to fix some execution order issues that a typical task based system can run into, to better increase system throughput. This could allow better debugging for some cases with the downside of having less ideal execution order.

While plausible, IMO, it would be impractical. A fiber-based task system fundamentally changes how tasks are created, dependencies are defined, and how synchronization happens. Attempting to abstract that all away would be very difficult IMO

[mdsitton]

Made a separate issue for the boost context items: #141

[mdsitton]

Some random musings, I wonder if a job library that can have fibers enabled and disabled through a compilation flag could be useful. Because technically fibers as I understand it mainly just get you an advantage to fix some execution order issues that a typical task based system can run into, to better increase system throughput. This could allow better debugging for some cases with the downside of having less ideal execution order.

While plausible, IMO, it would be impractical. A fiber-based task system fundamentally changes how tasks are created, dependencies are defined, and how synchronization happens. Attempting to abstract that all away would be very difficult IMO

Yeah i guess that's fair, i do still think it could be interesting though :)

[mdsitton]

139 - Probably just needs to run CI again and then merge. Though, better would be to upgrade nonius itself (instead of changing this fork)

About updating nonius i don't think there is anything to update that project seems fairly abandoned to me: https://github.com/libnonius/nonius/commits/devel

I mean you may get a few fixes but it doesn't look like there has been much development even since C++20 and the core issue in #139 ofc isn't resolved either: https://github.com/libnonius/nonius/blob/abdd1c4671e221daeed2c1b6ac557617f7b7a563/include/nonius/detail/complete_invoke.h%2B%2B#L46

Maybe another library could be used, here is a few i found with a quick look: https://github.com/martinus/nanobench https://github.com/DigitalInBlue/Celero https://github.com/google/benchmark

[mdsitton]

Actually further digging into things you already have catch2 integrated into the repo, and a few versions before what is currently in the project they integrated the nonius codebase into catch2 for writing benchmarks: https://github.com/catchorg/Catch2/releases/tag/v2.9.0

https://github.com/catchorg/Catch2/blob/devel/docs/benchmarks.md

So it could be possible to maybe make use of this and delete it all together just having the single dependency. Though catch2 will need to be updated as well due to the same issue in the benchmarking code: https://github.com/RichieSams/FiberTaskingLib/blob/master/third_party/catch2/include/catch2/catch.hpp#L6510

Upgrading to the latest version of catch will require this library to bump to C++14 minimum however but i suspect that really won't be a huge issue considering C++14 is almost 10 years old now.

[RichieSams]

Heh, I was just about to say that same. I recently updated catch2 to v3 for a work project, and I saw they added benchmarking support. So that's a probable avenue.

We'd only have to bump the tests to c++14 (which they might even already require 🤷‍♂️ ). Which I'm fine with. As you mentioned, 14 is almost 10 years old now

[RichieSams]

I'm going to close this issue, as I believe all topics brought up have been captured in their own issues. Feel free to open issues for anything I missed though