mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

fix read stderr during command execution #47

Closed ghost closed 3 years ago

ghost commented 4 years ago

Currently stderr is only read after stdout ended sending (= after the called bash command ended). With this change, stderr and stdout is read simultanously via two separate threads to make it possible that stderr is read before the command ended.

How to test this

Create file test-stderr.sh:

#/bin/bash

for number in {1..15}
do
    >&2 echo "$number ..."
    sleep 1
done

exit 0

Create file test-stdout.sh:

#/bin/bash

for number in {1..15}
do
    echo "$number ..."
    sleep 1
done

exit 0

When you set up a pipeline with a RunBash command calling test-stdout.sh, you will see the output during execution. When you do the same with calling test-stderr.sh, you will get the log output in mara only after the shell script ended.

With this pull request, this behavior will change so that stdout/stderr reading happens during command execution, so, test-stderr.sh will show the log output right away. You don't have to wait until the shell command ended before you can see stderr in the mara log.

martin-loetzsch commented 3 years ago

Thanks!