CWL-DAG Schedule bug

[seongwoo-jang7]

Describe the bug HI, I want to schedule using , CWL-airflow, but I don't think I can schedule it. Even though DAG included schedule_interval as a parameter, it appears as scheduling none in the Airflow Web UI. Is CWL Airflow not available for scheduling? Or did I do something wrong? Is it a bug? Please, help me. thanks!

Desktop (please complete the following information):

• OS: Mac OS
• Python version (from where installed: Anaconda, brew, etc): brew, python 3.8
• Installed python packages (run pip list or similar command) Package Version

  ---

  alembic 1.5.4 apache-airflow 2.0.0 apache-airflow-providers-ftp 1.0.1 apache-airflow-providers-google 1.0.0 apache-airflow-providers-http 1.1.0 apache-airflow-providers-imap 1.0.1 apache-airflow-providers-postgres 1.0.1 apache-airflow-providers-sqlite 1.0.1 apispec 3.3.2 argcomplete 1.12.2 attrs 20.3.0 Babel 2.9.0 bagit 1.8.1 CacheControl 0.11.7 cached-property 1.5.2 cachetools 4.2.1 cattrs 1.2.0 certifi 2020.12.5 cffi 1.14.5 chardet 3.0.4 click 7.1.2 clickclick 20.10.2 colorama 0.4.4 coloredlogs 15.0 colorlog 4.0.2 commonmark 0.9.1 connexion 2.7.0 croniter 0.3.37 cryptography 3.4.5 cwl-airflow 1.2.10 cwltest 2.0.20200626112502 cwltool 3.0.20200710214758 decorator 4.4.2 defusedxml 0.6.0 dill 0.3.3 dnspython 1.16.0 docker 5.0.0 docutils 0.16 email-validator 1.1.2 eventlet 0.30.1 Flask 1.1.2 Flask-AppBuilder 3.1.1 Flask-Babel 1.0.0 Flask-Caching 1.9.0 Flask-JWT-Extended 3.25.1 Flask-Login 0.4.1 Flask-OpenID 1.2.5 Flask-SQLAlchemy 2.4.4 flask-swagger 0.2.13 Flask-WTF 0.14.3 funcsigs 1.0.2 gevent 21.1.2 google-ads 7.0.0 google-api-core 1.26.0 google-api-python-client 1.12.8 google-auth 1.26.1 google-auth-httplib2 0.0.4 google-auth-oauthlib 0.4.2 google-cloud-automl 1.0.1 google-cloud-bigquery 2.8.0 google-cloud-bigquery-datatransfer 1.1.1 google-cloud-bigquery-storage 2.2.1 google-cloud-bigtable 1.7.0 google-cloud-container 1.0.1 google-cloud-core 1.6.0 google-cloud-datacatalog 0.7.0 google-cloud-dataproc 1.1.1 google-cloud-dlp 1.0.0 google-cloud-kms 1.4.0 google-cloud-language 1.3.0 google-cloud-logging 1.15.1 google-cloud-memcache 0.3.0 google-cloud-monitoring 1.1.0 google-cloud-os-login 1.0.0 google-cloud-pubsub 1.7.0 google-cloud-redis 1.0.0 google-cloud-secret-manager 1.0.0 google-cloud-spanner 1.19.1 google-cloud-speech 1.3.2 google-cloud-storage 1.36.0 google-cloud-tasks 1.5.0 google-cloud-texttospeech 1.0.1 google-cloud-translate 1.7.0 google-cloud-videointelligence 1.16.1 google-cloud-vision 1.0.0 google-crc32c 1.1.2 google-resumable-media 1.2.0 googleapis-common-protos 1.52.0 graphviz 0.16 greenlet 1.0.0 grpc-google-iam-v1 0.12.3 grpcio 1.35.0 grpcio-gcp 0.2.2 gunicorn 19.10.0 httplib2 0.19.0 humanfriendly 9.1 idna 2.10 importlib-metadata 1.7.0 importlib-resources 1.5.0 inflection 0.5.1 iso8601 0.1.14 isodate 0.6.0 itsdangerous 1.1.0 Jinja2 2.11.3 json-merge-patch 0.2 jsonmerge 1.8.0 jsonschema 3.2.0 junit-xml 1.9 lazy-object-proxy 1.4.3 libcst 0.3.17 lockfile 0.12.2 lxml 4.6.3 Mako 1.1.4 Markdown 3.3.3 MarkupSafe 1.1.1 marshmallow 3.10.0 marshmallow-enum 1.5.1 marshmallow-oneofschema 2.1.0 marshmallow-sqlalchemy 0.23.1 mistune 0.8.4 mypy-extensions 0.4.3 natsort 7.1.1 networkx 2.5.1 numpy 1.20.1 oauthlib 2.1.0 openapi-spec-validator 0.2.9 packaging 20.9 pandas 1.2.2 pandas-gbq 0.14.1 pendulum 2.1.2 pip 21.1.2 prison 0.1.3 proto-plus 1.13.0 protobuf 3.14.0 prov 1.5.1 psutil 5.8.0 psycopg2-binary 2.8.6 pyarrow 3.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.20 pydata-google-auth 1.1.0 Pygments 2.8.0 PyJWT 1.7.1 pyOpenSSL 19.1.0 pyparsing 2.4.7 pyrsistent 0.17.3 python-daemon 2.2.4 python-dateutil 2.8.1 python-editor 1.0.4 python-nvd3 0.15.0 python-slugify 4.0.1 python3-openid 3.2.0 pytz 2020.5 pytzdata 2020.1 PyYAML 5.4.1 rdflib 4.2.2 rdflib-jsonld 0.5.0 requests 2.23.0 requests-oauthlib 1.1.0 rich 9.2.0 rsa 4.7 ruamel.yaml 0.16.5 schema-salad 7.1.20210518142926 setproctitle 1.2.2 setuptools 57.0.0 shellescape 3.4.1 six 1.15.0 SQLAlchemy 1.3.20 SQLAlchemy-JSONField 1.0.0 SQLAlchemy-Utils 0.36.8 swagger-ui-bundle 0.0.8 tabulate 0.8.7 tenacity 6.2.0 termcolor 1.1.0 text-unidecode 1.3 thrift 0.13.0 tornado 6.1 typing-extensions 3.7.4.3 typing-inspect 0.6.0 tzlocal 1.5.1 unicodecsv 0.14.1 uritemplate 3.0.1 urllib3 1.25.11 websocket-client 1.0.1 Werkzeug 1.0.1 WTForms 2.3.3 zipp 3.4.0 zope.event 4.5.0 zope.interface 5.2.0

[michael-kotliar]

Hi @seongwoo-jang7 , Thank you for submitting an issue. In the current implementation of CWL-Airflow we intentionally set "schedule_interval": None as we envision CWLDAGs to be externally triggered. Could you please provide an explanation of your use case, so we can find a proper solution?

https://github.com/Barski-lab/cwl-airflow/blob/8aa26f6a50a8ea15bb8d544225f2dfe09668a5f3/cwl_airflow/extensions/cwldag.py#L106

[seongwoo-jang7]

Hi, @michael-kotliar Thanks for your coments.

I am using CWL-airflow for the purpose of proceeding with ETL work. So I asked because I needed scheduling. I understand your intentions and can you make to able scheduling? can schedule it, and the external trigger.

And i have an additional question, how can I use the airflow operator? For example, I want to use a big query operator among airflow operators to put data in the middle of the cwl step. Thanks!

[michael-kotliar]

Hi, @seongwoo-jang7, sorry for the late response.

If you install CWL-Airflow from the latest commit on the master branch, you will be able to provide your own schedule_interval in CWLDAG constructor. Let me know if something doesn't work as expected.

As for the calling big query operator, in between CWL workflow steps execution, it's currently not that easy to implement, as it will change the structure of DAG. Our concept was to load DAG from CWL file so it completely depends on it. However, there is a solution. Have you considered using ExternalTaskSensor?

[seongwoo-jang7]

I've considered the External Sensor, but it is not i want. I'll try schedule_interval again and open the issue if it doesn't work. Thank you for your comments!