search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
185 stars 32 forks source link

Update cwl-airflow for better usability #8

Closed michael-kotliar closed 6 years ago

michael-kotliar commented 6 years ago
  1. Upload cwl-airflow on pip
  2. Update cwl-airflow with:
    • init - search for airflow.cfg, update airflow.cfg (instead of calling post_install.sh from setup.py)
      • don't call airflow initdb, check if airflow.cfg is present
      • don't create any folders
    • test - to download sample workflow and put it in the correct folders
  3. Make jobdispatcher.py to create all necessary folders according to [cwl] section from airflow.cfg. Do not fail, if some folder is not present
  4. Don't fail on Exception in cwl_dag.py 
    except Exception as ex:
    print "FAIL exception: ", str(ex)
    shutil.move(fn, os.path.join('/'.join(fn.split('/')[0:-2]), 'fail'))
  5. Remove #!/usr/bin/env python from files or make sure that it's used properly
michael-kotliar commented 6 years ago

Folders for workflows and jobs should be created when init is called. If we create them from JobDispatcher node, we should at least once run airflow scheduler, which is not correct, as long as we need these folders to put jobs and workflows into them before we run airflow scheduler