search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
185 stars 32 forks source link

cwl-airflow run command output file was not in configured output folder #10

Closed Huiziryuu closed 6 years ago

Huiziryuu commented 6 years ago

I tested command line function by ruuning cwl-airflow run /home/hui/ariflow/cwlworkflows/example.cwl /home/hui/airflow/cwl/jobs/new/example-job.yaml

the command was run in the folder level '~/airflow'. test files are cwlExample.zip

The output text file is in '\~/airflow' folder where I invoked the command, shouldn't it be in the airflow.cfg configured ''\~/airflow/cwl/ouput'' folder?

michael-kotliar commented 6 years ago

If you use cwl-airflow run --ignore-def-outdir the output files will be placed into the output folder set in Airflow configuration file. Otherwise they will be stored in the same directory where you run cwl-airflow run. I would recommend to run workflows through airflow scheduler to use all of the features available from Airflow. cwl-airflow run is mostly used for CWL conformance tests

Huiziryuu commented 6 years ago

Hi, @michael-kotliar Thanks. I got airflow scheduler runs. And I can see the execution logs from UI, however the flow's ID is empty. How should I configure the ID's for it. Would it be possible to see the flow DAG graph through airflow UI?

michael-kotliar commented 6 years ago

Hi @Huiziryuu When DAG generated from cwl and job files is running it's almost the same as any other Airflow DAG, so all of the items from standard Airflow UI are available by default. The only difference is that you cannot restart this DAG from UI interface, but it's related to the question I've already answered here #11 If for some reasons your AIrflow UI doesn't show some of the tabs, make sure that your cwl file is correct, properly loaded and docker is available (if it's used in a workflow). If you run an example workflow from README and have the same problems, I will be happy to solve these problems.

michael-kotliar commented 6 years ago

As of v1.0.13 output_folder is set directly in the Job file or as -o optional parameter when submitting the job.