search

ESGF / esgf-compute-wps

WPS API for ESGF Compute Working Team
13 stars 10 forks source link

Description of CWL Airflow use case #146

Closed tomLandry closed 5 years ago

tomLandry commented 6 years ago

Hello CWT! We have projects at OGC that calls for implementation of Workflows using CWL runners. We plan to use Toil, but I would like to know how you plan to use Airflow. Maybe there is a good opportunity for interoperability here.

Any sequence diagrams on this? Components diagrams?

jasonb5 commented 6 years ago

Right now we're just investigating moving from Celery canvas to Airflow. The plan for the moment is to design an Airflow DAG that represents a single operation. This operation DAG will become a sub dag when a workflow is requested. This is just preliminary, so I don't have any diagrams put together yet. As for CWL, I found cwl-airflow, so there may be an opportunity there.

tomLandry commented 6 years ago

Well we're on a good page for CWL, and we can manage Celery too. Our CWL scripts wraps docker images (applications).