search

vatlab / sos

SoS workflow system for daily data analysis
http://vatlab.github.io/sos-docs
BSD 3-Clause "New" or "Revised" License
274 stars 45 forks source link

Using asyncio for concurrent execution #1491

Open BoPeng opened 2 years ago

BoPeng commented 2 years ago

SoS went very far in allowing multiple tasks to be executed on the same process. Mixing it with zmq made the executor very complicated. In essence, we divde the execution into generators and "yield" to other tasks when one tasks is waiting for something. To achieve that, we have a whole set of stuff to "switch context".

This is essentially the re-invention of Python co-routine (to be fair, we started this before Python 3.7) so it may be much easier to re-implement these using coroutine. The problem is that this is a lot of work and we do not really want to break things that are working at this point.

BoPeng commented 2 years ago

The major technical problem has been solved using contextvars module

https://stackoverflow.com/questions/71930608/context-switching-between-python-coroutines