Question: Scatter Support

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support

https://barski-lab.github.io/cwl-airflow

Apache License 2.0

183 stars 33 forks source link

Question: Scatter Support #48

Open dagrooms52 opened 3 years ago

dagrooms52 commented 3 years ago

According to this release, this project now supports "simple scatter" in CWL workflows. What is the definition of "simple scatter?" Can I expect Airflow to process Scatter tasks concurrently using this project?

michael-kotliar commented 3 years ago

Hello, Sorry for the late response. "Simple scatter" support means that you can run CWL workflows with scatter, but you can't see them properly parsed into the DAG. Also, the scattered tasks will be run sequentially as at the moment I can't control them from inside Airflow. I'm working on making Scatter work in the proper for Airflow way.