airflow dags which use anyway-kubectl-exec should print output while they are running

data-for-change / anyway-etl

MIT License

0 stars 7 forks source link

airflow dags which use anyway-kubectl-exec should print output while they are running #13

Closed OriHoch closed 3 years ago

OriHoch commented 3 years ago

reproduction steps

run an airflow dag which execute via anyway-main pod using anyway-kubectl-exec, for example: https://airflow.anyway.co.il/log?dag_id=fill-infographics-cache-for-road-segments&task_id=fill-infographics-cache-for-road-segments&execution_date=2021-10-06T08%3A23%3A24.570342%2B00%3A00
check the execution log in airflow while the dag is running

expected

should show the output immediatley as it is printed

actual

prints show only after execution ends

OriHoch commented 3 years ago

looks like output is being printer as it comes in, I added a simple test dag and it works - https://dev-airflow.anyway.co.il/tree?dag_id=test-anyway-kubectl-exec

this is most likely a problem with logging configuration, it looks like there isn't any centralized logging configuration. I recommend to just use print statements for CLI tasks. Otherwise, sort out the logging configuration.

OriHoch commented 3 years ago

@atalyaalon