support async Python actions

wahuneke commented 2 years ago

Motivations

Two motivations for this (sizeable) request:

it may provide a simpler route to parallel action support (which is currently unsupported in Windows and has limitations elsewhere)
allow for integration against async libraries (e.g. aiodocker)

Usage

1) Declaring a Python async function as a task's action would result in that task's action being pushed onto the event loop. Additional plumbing would be added to the loop as well in order to chain dependent tasks in downstream 2) Activating a global configuration option would cause doit to automatically run all shell command blocks using an async shell exec adapter such as Python's async subprocess feature

The work

I have not looked at doit code yet, but would consider volunteering for some / all of this effort if more-experienced contributors believe it is worth pursuing.

I did some quick searches through PRs and existing issues and did not see any discussion of Python's async faculties... Sorry if I missed something or if this is already asked and answered

schettino72 commented 2 years ago

This would definitely be a nice addition.

The code you are looking for is at runner.py. You can see default implementation and parallel multiprocess/thread.

Actually I have done some experimentation related to this a while ago (in a different project)... I want to move away from using the multiprocess module altogether.

I guess a first step would be to just support async actions (run just one at time).

As of today you can specify the parallel mechanism only per run (i.e. all tasks a threads or process). Ideally would be able to group tasks and run them on different parallel strategies with master as async code.

pydoit / doit