cwl-airflow setpu fails

[Huiziryuu]

When I install cwl-airflow, the installation fails on step pip install --user .

the error message is: `Processing /home/hui/cwl-airflow/cwl-airflow Collecting cwltool==1.0.20180116213856 (from cwl-airflow==1.0.20180129042848) Using cached cwltool-1.0.20180116213856-py2.py3-none-any.whl Collecting jsonmerge (from cwl-airflow==1.0.20180129042848) Using cached jsonmerge-1.4.0.tar.gz Collecting mysql-python>=1.2.5 (from cwl-airflow==1.0.20180129042848) Using cached MySQL-python-1.2.5.zip Complete output from command python setup.py egg_info: sh: 1: mysql_config: not found Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-KyPi8b/mysql-python/setup.py", line 17, in metadata, options = get_config() File "/tmp/pip-build-KyPi8b/mysql-python/setup_posix.py", line 43, in get_config libs = mysql_config("libs_r") File "/tmp/pip-build-KyPi8b/mysql-python/setup_posix.py", line 25, in mysql_config raise EnvironmentError("%s not found" % (mysql_config.path,)) EnvironmentError: mysql_config not found

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-KyPi8b/mysql-python/ `

[Huiziryuu]

BTW, I don't need mysql at this point, is it possible to just install this tool running over cwl rquired script and config files?

[michael-kotliar]

Hi @Huiziryuu Try to run

sudo apt-get install libmysqlclient-dev

It should fix a problem of missing mysql_config

[Huiziryuu]

@michael-kotliar thanks! Now I'm using your new pip command to install the package, I get prompt message ask me to uninstall old non-apache version Airflow. Can I install cwl-airflow as independent package? In case in the future if it's in production, and only cwl-airflow needs to upgrade. :-)

[Huiziryuu]

error message: `Collecting cwl-airflow Downloading cwl-airflow-1.0.4.tar.gz Collecting cwltool==1.0.20180116213856 (from cwl-airflow) Downloading cwltool-1.0.20180116213856-py2.py3-none-any.whl (353kB) 100% |████████████████████████████████| 358kB 1.7MB/s Collecting jsonmerge (from cwl-airflow) Downloading jsonmerge-1.4.0.tar.gz Collecting mysql-python>=1.2.5 (from cwl-airflow) Downloading MySQL-python-1.2.5.zip (108kB) 100% |████████████████████████████████| 112kB 3.3MB/s Requirement already satisfied: ruamel.yaml<0.15 in /usr/local/lib/python2.7/dist-packages (from cwl-airflow) Collecting apache-airflow==1.8.2 (from cwl-airflow) Downloading apache-airflow-1.8.2.tar.gz (2.3MB) 100% |████████████████████████████████| 2.3MB 359kB/s Complete output from command python setup.py egg_info: An earlier non-apache version of Airflow was installed, please uninstall it first. Then reinstall.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-JKmIUS/apache-airflow/ `

[michael-kotliar]

@Huiziryuu I think your error message is not directly related to cwl-airflow. Probably, you have already installed airflow<=1.8.0 which is not apache-airflow. You can check it with pip show airflow. To be able to use apache-airflow==1.8.2 (which is required by cwl-airflow), you should first remove the old airflow package with pip uninstall airflow. cwl-airflow is an extension to the main Airflow functionality. It works like a parser and executor of CWL workflows and cannot function without apache-airflow. If you want to use airflow instead of apache-airflow, you may try to get the source code of cwl-airflow and update setup.py with the package version you need, but it's not recommended as it wasn't designed in such a way.

[Huiziryuu]

@michael-kotliar thanks for explanation. found another problem that, if user install cwl-airflow in a complete clean server from scratch, the installation fails, the error was got rid of by installing the git command. Should this be reminded in the environment preparation? Error log: Collecting cwl-airflow Downloading cwl-airflow-1.0.4.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-so6_Yc/cwl-airflow/setup.py", line 52, in version=get_version(), File "/tmp/pip-build-so6_Yc/cwl-airflow/setup.py", line 38, in get_version version = get_git_tag() # try to get version info from the closest tag File "/tmp/pip-build-so6_Yc/cwl-airflow/setup.py", line 11, in get_git_tag return subprocess.check_output(['git', 'describe', '--contains']).strip() File "/usr/lib/python2.7/subprocess.py", line 567, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 711, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-so6_Yc/cwl-airflow/

[michael-kotliar]

@Huiziryuu Thanks a lot! I've updated Readme. Git is used to set a version of a package from git tag or git commit timestamp

[michael-kotliar]

As of v1.0.13 libmysqlclient-dev is not required anymore