omnilib / aioitertools

itertools and builtins for AsyncIO and mixed iterables
https://aioitertools.omnilib.dev
MIT License
240 stars 24 forks source link

Allow slightly unbalanced consumption in tee #56

Open dcsommer opened 3 years ago

dcsommer commented 3 years ago

The implementation of aioitertools.itertools.tee currently has the caveat "all iterators are dependent on the first iterator." This caveat can cause periodic stalls in my pipeline where the consumers all consume at roughly the same speed, with small imbalances.

A small refactor of tee could remove this limitation if the producer was a separate scheduled task (instead of the first consumer). Would you be open to a pull request for a change along these lines?

amyreese commented 3 years ago

A PR would be great, but I would like to see a couple constraints:

By default, this would still cause consumption to be limited by the slowest iterator, but no longer dependent on the first iterator specifically.