Open leo-schick opened 2 years ago
@jankatins @gathineou @martin-loetzsch would like to get your feedback here what you guys think about this idea
I know about the option mara_pipelines.config.bash_command_string
but this wasn't enogh for me because I need to be able to execute multiple execution contexts on the same server and do not want to use multiple mara config files.
It would be nice to have a SQL execution context as well. This context would then run e.g. the ExecuteSQL
command via the python DB API (see https://github.com/mara/mara-db/pull/71).
The development would require mor refactoring than the current implementation which just patches the way batch commands are executed.
Currently mara pipelines are always executed locally. But I would like to have an option to execute it sometimes somewhere else e.g. in another environment where other ressources are closer available.
The idea
So I came up with the idea about execution contexts. Here is the rough idea:
The current idea is to support the following execution context:
BashExecutionContext
- local bash (this is the current default behavior)SshBashExecutionContext
- remote bash execution via sshDockerExecutionContext
- docker exec with optional start/stop of a containerPossible other options (Out of scope)
This concept could be extended in the future to add other options like:
These ideas are just noted here and are out of scope for this issue.
Blueprint for the ExecutionContext base class