new approach

[ddurham2]

@saghul, I'm looking into the 3.11, 3.12 support and it's becoming obvious that the cpython engine changes every few releases in a way that breaks fibers.c/h (3.7 needed changes, 3.11 will need changes, and 3.12 will need still more changes)

I read your why.rst doc and sympathize with the reasons you initially created fibers. And I noticed that it compares itself to greenlet's behaviors (a package, incidentally, that I was unaware of when I initially went searching for a fibers package). It does nearly the same thing but with some specific set behaviors.

But I'm wondering if it might be worthwhile to make fibers into simply a pure python wrapper around the greenlet package? I haven't explored it fully yet, but I would be willing to explore how all the issues could be worked around, and keep the same behavior of your package, verified via the unit tests.

My only question is about performance: Will it perform horribly worse, or will it be quite acceptable? We could add a benchmark unit test to compare the two approaches.

Also you mention that you went with stacklet because it performs the save and restore operations within the asm which is better because of compiler behaviors. But I'd think it okay to just let the greenlet maintainers worry about those issues since it's an active and well-maintained library (w/ new release only 5 days ago).

If possible, even if there's just a slight performance penalty, wouldn't it be worth it to avoid the hassle of always chasing inevitable changes to the internals of cpython?

Thoughts?

[saghul]

Hey there! Thanks for reigniting the project!

But I'm wondering if it might be worthwhile to make fibers into simply a pure python wrapper around the greenlet package? I haven't explored it fully yet, but I would be willing to explore how all the issues could be worked around, and keep the same behavior of your package, verified via the unit tests.

I'm not sure it would be possible to do that while maintaining all of the goals that this project sat out to accomplish. One of the main things I disliked from Greenlet was the behavior under GC, and I don't think we can fix that.

If possible, even if there's just a slight performance penalty, wouldn't it be worth it to avoid the hassle of always chasing inevitable changes to the internals of cpython?

I've maintained C Python libraries since around Python 2.5. By far the largest set of changes came with the migration to Python 3, and now that it's over, it's usually minor stuff. I'm not worried about the long term maintenance, since it's attainable.

This package would cease to have a purpose if it just wraps greenlet. Anyone can wrap greenlet to have an API they like, can't they? :-)

[ddurham2]

Hey! Awesome! Glad it merged smoothly and everything is working. I see you did some additional cleanup.

BTW- I'm not sure if that pyfibers, continuation-based, implementation still works. Is that expected to work if running under pypa? I didn't try that.

[ddurham2]

I'm not sure it would be possible to do that while maintaining all of the goals that this project sat out to accomplish. One of the main things I disliked from Greenlet was the behavior under GC, and I don't think we can fix that. I took a stab at it this evening and I did get all the unit tests passing (except the dict tests which are ancillary, but I will address if we continue this discussion).

The short and sweet implementation is here. Full comparison here.

So, I didn't look closely at the gc unit test, but I'm wondering if there's a difference, do the unit tests specifically test for it to behave as you expect? I'd like to know more about the GC to understand. I have a general working knowledge of the 2 phase GC. But.. Some questions:

1. Under what circumstances would this difference in behavior be seen?
2. Does it have to do with whether a fiber is possibly reactivated under GC?
3. Greenlet, I believe, is careful to reactivate a fiber via an exception just to ensure there any critical cleanup routines will have run (e.g. closing files, exit's for context managers, etc.) I think that is vitally important. Does Fibers differ with that behavior? But maybe that has nothing to do with the behavior in queston. I've maintained C Python libraries since around Python 2.5. By far the largest set of changes came with the migration to Python 3, and now that it's over, it's usually minor stuff. I'm not worried about the long term maintenance, since it's attainable.

I've maintained C Python libraries since around Python 2.5. By far the largest set of changes came with the migration to Python 3, and now that it's over, it's usually minor stuff. I'm not worried about the long term maintenance, since it's attainable.

While that may be true, it's also true that Fiber's hasn't had a release since 2017. Anything depending on it for production really couldn't use it (as I am attempting to do). I assume it hasn't had a release because the maintenance isn't free. The GH action changes at least makes the release process closer to zero friction. But the next is maintaining the code against python versions, and dealing with bugs whenever they are reported.

This package would cease to have a purpose if it just wraps greenlet. Anyone can wrap greenlet to have an API they like, can't they? :-) I'd suggest it could still serve a purpose because it offers a bit cleaner interface along with all the unit tests to ensure that behaviors remain unchanged from release to release.

If Fibers' implementation can be greatly simplified by leveraging a project like greenlet which is dedicated to the large effort of keeping up with py and fixing bugs as they are reported, then why make things harder for yourself? Create the simpler wrapper with the interface and specific behaviors you want, and let the community do the heavy lifting while you solve the higher level problem you (and others) needed Fibers for in the first place. No?

Hey, but I get the fun of building the low-level lib yourself and seeing it work. We love to be tinkerers. If that's all this lib is, then people (like myself) need to know that so they can judge whether it's safe to build production systems on, and whether future releases can be expected or not.

Thoughts?