search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
185 stars 32 forks source link

CWLJobDispatcher, {task}, CWLJobGatherer not running after triggering with Airflow API #68

Closed kokleong9406 closed 3 years ago

kokleong9406 commented 3 years ago

Hi, when I tried to trigger a {dag_id} using the standard Airflow API [(https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag_runs):]()

curl -fsSL -X POST -d '{"execution_date": "2021-01-13T15:00:00Z", "conf": {"job": {"message": "ABC"}}}' 'http://localhost:8080/api/v1/dags/{dag_id}/dagRuns' -H 'content-type: application/json' --user "admin:admin"

I found that the tasks: CWLJobDispatcher, echo, CWLJobGatherer are not running, as shown below when I looked at my {dag_id} status in the airflow webserver. You can see that all the three tasks are not Green, but blank. I have also checked that I have UNPAUSE my dag.

image

(Note: I have no issue using the CWL-Airflow API with regards to tasks not running)

The reason I am resorting to using the Airflow API instead of CWL-Airflow API is because Airflow API allows me to authenticate username:password in my curl command, and setup role-based access control for different roles/users. Am I missing something here?

kokleong9406 commented 3 years ago

Silly me, I found out the reason! When triggering the API, I should not include "execution_date". My problem is solved using below curl command without the "execution_date"

curl -fsSL -X POST -d '{"conf": {"job": {"message": "ABC"}}}' 'http://localhost:8080/api/v1/dags/{dag_id}/dagRuns' -H 'content-type: application/json' --user "admin:admin"

michael-kotliar commented 3 years ago

Ok:) Let me know if you need help with any other questions:)