ezmsg-org / ezmsg

Pure-Python DAG-based high-performance SHM-backed pub-sub and multi-processing pattern
https://ezmsg.readthedocs.io/en/latest/
MIT License
15 stars 6 forks source link

Refactor data transformers to use class objects instead of generator methods #158

Open cboulay opened 1 week ago

cboulay commented 1 week ago

Generator methods are problematic for a few reasons. For example, they can't be pickled which means they can't be moved across processes using ezmsg's built-in multiprocessing. Additionally, they can't be initialized from scratch with a pre-existing state, which precludes them from use in processors in cloud computing paradigms:

We have decided that we are going to replace the generator methods with picklable objects and with the possibility of initializing them with a pre-existing state. The exact design is yet to be determined so we will go through a design phase which will be informed by some attempts to implement the initial design.

griffinmilsap commented 1 week ago

@pperanich would you have time in the next week or two to figure out a Protocol proposal here?