search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
183 stars 33 forks source link

Question: Can i use airflow operators with CWL-Airflow? #63

Open Amitg1 opened 3 years ago

Amitg1 commented 3 years ago

Can I use airflow operators with CWL-Airflow? or only cmdtools like in CWL?

if so, how I am expected to push new operators or cmdtools to Airflow?

Thanks! great job.

michael-kotliar commented 3 years ago

Hello, CWL format is more suitable for running command-line tools. And if you are using Docker containers each step will be completely isolated from others, so it's not an easy task to run standard Airflow operator from inside CWL step. I would recommend using something like ExternalTaskSensor for that purpose.

Would you mind describing your use case? Which Airflow Operators are you planning to use and after which workflow steps? Perhaps, this is some common scenario that I'm not aware of and that is worth implementing in CWL-Airflow natively.