mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

add execution contexts #69

Open leo-schick opened 2 years ago

leo-schick commented 2 years ago

In WIP feature discussion in #68

leo-schick commented 2 years ago

Current status

the BashExecutionContext and SshBashExecutionContext works, but logging should be optimized

Sample app

local_config.py

from mara_pipelines.contexts.bash import BashExecutionContext, SshBashExecutionContext
patch(mara_pipelines.config.execution_contexts)(lambda: {
    'bash': BashExecutionContext(),
    'srv-mara-2': SshBashExecutionContext(
        host='srv-mara-2'
    ),
})

test.py

from mara_pipelines.pipelines import Pipeline, Task
from mara_pipelines.commands.bash import RunBash

test_pipeline = Pipeline(
    id='test',
    description="",
    context='srv-mara-2')

test_pipeline.add(
    Task(id='ls',
         description="",
         commands=[
             RunBash('ls -l')
         ]))

test_pipeline.add(
    Task(id='ls2',
         description="",
         commands=[
             RunBash('ls -l .venv')
         ],
         context='bash'),
    upstreams=['ls'])

Live execution output

image

Output after execution:

image

Open tasks

leo-schick commented 2 years ago

Currently known issue: when a execution context is used which is not defined in the configuration the pipeline execution does not throw any message and just hangs