search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
183 stars 33 forks source link

Redirect CommandLineTool output to default task log file #61

Open francescobianco opened 3 years ago

francescobianco commented 3 years ago

I want collect all generated output by every task into standard Airflow task log file, generally is ./logs/{dag_id}/{task_id}/{run_stamp}/1.log

Furthermore I would get all logs for every task into single file, I want collapse all log into file 1.log and not into 1.log.cwl

Screenshot from 2021-06-18 03-24-25

michael-kotliar commented 3 years ago

Hi @francescobianco Sure, it's easy to do. I intentionally patch airflow.cfg to split the logs into two files to keep airflow and cwl logs separately. You can see how I do it here https://github.com/Barski-lab/cwl-airflow/blob/9c2c8522c0969458915123e501591ebf15cc69f5/cwl_airflow/components/init/config.py#L105

If you want to to keep both of the logs in the same file, set the default value for logging_config_class in airflow.cfg.

Let me know if you have any other questions.