search

mindee / tawazi

A DAG Scheduler library written in pure python
https://mindee.github.io/tawazi/
Apache License 2.0
70 stars 5 forks source link

[FEATURE] async support #209

Open DeoLeung opened 7 months ago

DeoLeung commented 7 months ago

Is your feature request related to a problem? Please describe. dag execute async functions

Describe the solution you'd like could decorate async function the decorator support passing in pool executor in case for running sync

bashirmindee commented 6 months ago

Can you please describe the purpose of doing this ? The goal of tawazi is to paralellize the execution of sync code. Are you forced to use async code ?

DeoLeung commented 6 months ago

we use fastapi async mode, each node in the execution graph just a http request to other service. using async will be of great help.

bashirmindee commented 6 months ago

this feature is not in the roadmap sadly. But if you transform each node to sync, tawazi wil run each node in a separate thread. This would be helpful because your http requests won't block the Python interpreter. This should achieve the results you want.

However I still don't understand why you want to use tawazi in your case... if your code is already completely async, you don't need tawazi... However, if some parts of your code are async and others are sync tawazi is useful for you.

bashirmindee commented 1 month ago

we are working on async support here: https://github.com/mindee/tawazi/pull/217

It doesn't support async methods yet but transforms sync functions / ExecNodes into awaitables. Can this be useful to your use case ?

Currently we don't have the incentive to do async support. It could be implemented with ease now though since the code is now adapted to async