Use `asyncio` for async operations in Runners

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

https://kedro.org

Apache License 2.0

10.03k stars 906 forks source link

Use `asyncio` for async operations in Runners #4289

Open merelcht opened 4 weeks ago

merelcht commented 4 weeks ago

Description

We can refactor the code in Task that handles asynchronous loading and saving of data to use asyncio.

Context

https://github.com/kedro-org/kedro/pull/4206#pullrequestreview-2392331530

astrojuanlu commented 3 weeks ago

The idea is to refactor this method here:

https://github.com/kedro-org/kedro/blob/a1fae5018f35243a5e49a54a9dd3223b2c4ea743/kedro/runner/task.py#L192-L198

Possible inspiration:

https://www.roguelynn.com/words/asyncio-we-did-it-wrong/
https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
https://docs.python.org/3/library/asyncio-task.html#task-groups (added in version 3.11, ugh)
https://anyio.readthedocs.io/en/stable/tasks.html

We set an appetite of 1 sprint but we might need another one to wrap it up.