Closed azzaea closed 3 years ago
Hi @azzaea , Thanks for submitting an issue. It looks like the problem is in this line.
pkg_resources.ContextualVersionConflict: (colorlog 4.7.2 (/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages), Requirement.parse('colorlog==4.0.2'), {'apache-airflow'})
Try to run pip3 install -U colorlog==4.0.2
and see if it resolves an issue. Meanwhile, I will check why apache-airflow complains about the wrong version of colorlog
when it's explicitly set to be colorlog==4.7.2
by airflow developers.
Thank you @michael-kotliar , much appreciated.
I'm not sure this fixes it, as the logs now indicate loads of compatibility issues everywhere. I used the same virtual environment as before, and the logs are below.
$ source pythonenv/cwlairflow/bin/activate
$ pip3 install -U colorlog==4.0.2
Collecting colorlog==4.0.2
Downloading colorlog-4.0.2-py2.py3-none-any.whl (17 kB)
ERROR: apache-airflow 2.0.0 has requirement requests<2.24.0,>=2.20.0, but you'll have requests 2.25.1 which is incompatible.
ERROR: apache-airflow 2.0.0 has requirement tzlocal<2.0.0,>=1.4, but you'll have tzlocal 2.1 which is incompatible.
ERROR: apache-airflow-providers-http 2.0.0 has requirement apache-airflow>=2.1.0, but you'll have apache-airflow 2.0.0 which is incompatible. Installing collected packages: colorlog Attempting uninstall: colorlog Found existing installation: colorlog 4.7.2
Uninstalling colorlog-4.7.2:
Successfully uninstalled colorlog-4.7.2 Successfully installed colorlog-4.0.2
$
$ pip3 install cwl-airflow --constraint "https://raw.githubusercontent.com/Barski-lab/cwl-airflow/master/packaging/constraints/constraints-3.8.txt"
:
Requirement already satisfied: defusedxml==0.6.0 in ./pythonenv/cwlairflow/lib/python3.8/site-packages (from -c https://raw.githubusercontent.co
m/Barski-lab/cwl-airflow/master/packaging/constraints/constraints-3.8.txt (line 115)) (0.6.0)
ERROR: apache-airflow 2.0.0 has requirement colorlog==4.0.2, but you'll have colorlog 4.7.2 which is incompatible.
ERROR: apache-airflow 2.0.0 has requirement requests<2.24.0,>=2.20.0, but you'll have requests 2.25.1 which is incompatible.
ERROR: apache-airflow 2.0.0 has requirement tzlocal<2.0.0,>=1.4, but you'll have tzlocal 2.1 which is incompatible.
ERROR: apache-airflow-providers-http 2.0.0 has requirement apache-airflow>=2.1.0, but you'll have apache-airflow 2.0.0 which is incompatible.
Installing collected packages: colorlog
Attempting uninstall: colorlog
Found existing installation: colorlog 4.0.2
Uninstalling colorlog-4.0.2:
Successfully uninstalled colorlog-4.0.2
Successfully installed colorlog-4.7.2
$
$ cwl-airflow init
Traceback (most recent call last):
File "/home/azza/pythonenv/cwlairflow/bin/cwl-airflow", line 15, in <module>
sys.exit(main(sys.argv[1:]))
File "/home/azza/pythonenv/cwlairflow/bin/cwl-airflow", line 9, in main
args = parse_arguments(argsl)
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 257, in parse_arguments args, _ = get_parser().parse_known_args(argsl)
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 66, in get_parser version=get_version(),
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/helpers.py", line 211, in get_version pkg = pkg_resources.require("cwl_airflow")
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (tzlocal 2.1 (/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages), Requirement.parse('tzlocal<
2.0.0,>=1.4'), {'apache-airflow'})
$
$ export AIRFLOW_HOME=~/airflow
$ AIRFLOW_VERSION=2.1.0
$ PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
$ CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
$ pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
:
Successfully installed Babel-2.9.1 Flask-AppBuilder-3.3.0 Flask-Caching-1.10.1 Flask-JWT-Extended-3.25.1 Flask-SQLAlchemy-2.5.1 Markdown-3.3.4 P
ygments-2.9.0 SQLAlchemy-1.3.24 SQLAlchemy-Utils-0.37.2 alembic-1.6.2 apache-airflow-2.1.0 argcomplete-1.12.3 cattrs-1.6.0 cffi-1.14.5 colorlog-
5.0.1 croniter-1.0.13 cryptography-3.4.7 defusedxml-0.7.1 dill-0.3.1.1 docutils-0.17.1 gunicorn-20.1.0 marshmallow-3.12.1 numpy-1.20.3 openapi-s
pec-validator-0.3.0 pandas-1.2.4 python-daemon-2.3.0 pytz-2021.1 six-1.16.0 tabulate-0.8.9 zipp-3.4.1
$
$ cwl-airflow init
/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The hide_sensitive_variable_fields
option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your
config.
/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The default_queue option in [celery
] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
Traceback (most recent call last):
File "/home/azza/pythonenv/cwlairflow/bin/cwl-airflow", line 15, in <module>
sys.exit(main(sys.argv[1:]))
File "/home/azza/pythonenv/cwlairflow/bin/cwl-airflow", line 9, in main
args = parse_arguments(argsl)
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 257, in parse_arguments
args, _ = get_parser().parse_known_args(argsl)
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 66, in get_parser
version=get_version(),
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/cwl_airflow/utilities/helpers.py", line 211, in get_version
pkg = pkg_resources.require("cwl_airflow")
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (apache-airflow 2.1.0 (/home/azza/pythonenv/cwlairflow/lib/python3.8/site-packages), Requirement.parse(
'apache-airflow==2.0.0'), {'cwl-airflow'})
Hi @azzaea,
In the latest CWL-Airflow 1.2.11 all dependency issues should be fixed. I've tested it on Ubuntu 18.04/20.04 with Python 3.6.8/3.7.9/3.8.10.
Let me know if you still have problems with CWL-Airflow installation and running. Michael
Feel free to reopen this issue if the problem still exists.
Hi @michael-kotliar ,
Thank you for the update. I'm happy to confirm the new version seems to install fine on my laptop, despite showing a deprecation warning (messages were omitted for brevity):
$ python3 -m venv ~/pythonenv/cwlair
$ source ~/pythonenv/cwlair/bin/activate
$ export PYTHON_VERSION=`python3 --version | cut -d " " -f 2 | cut -d "." -f 1,2`
$ pip3 install cwl-airflow --constraint https://raw.githubusercontent.com/Barski-lab/cwl-airflow/master/packaging/constraints/constraints-${PYTHON_VERSION}.txt
$ cwl-airflow init
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
However, running the example pipeline from the Quick start did not go as smoothly as one hopes for:
$ git clone https://github.com/datirium/workflows.git --recursive
$ airflow users create \
> --username admin \
> --firstname firstname \
> --lastname lastname \
> --role Admin \
> --email azzaea@gmail.com
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2021-10-10 19:16:24,223] {logging_config.py:46} INFO - Successfully imported user-defined logging config from cwl_airflow.config_templates.airflow_local_settings.DEFAULT_LOGGING_CONFIG
[2021-10-10 19:16:24,269] {manager.py:788} WARNING - No user yet created, use flask fab command to do it.
Password:
Repeat for confirmation:
Admin user admin created
$ # The below commands were all run on separate terminals (in parallel), after having the cwl environment activated
$ # airflow scheduler
$ # airflow webserver
$ # cwl-airflow api
$ # And, in the terminal where I cloned the workflow:
$ cwl-airflow test --suite workflows/tests/conformance_tests.yaml --range 1
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
Traceback (most recent call last):
File "/home/azza/pythonenv/cwlair/bin/cwl-airflow", line 15, in <module>
sys.exit(main(sys.argv[1:]))
File "/home/azza/pythonenv/cwlair/bin/cwl-airflow", line 9, in main
args = parse_arguments(argsl)
File "/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 299, in parse_arguments
assert_and_fix_args(args)
File "/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 265, in assert_and_fix_args
assert_and_fix_args_for_test(args)
File "/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/cwl_airflow/utilities/parser.py", line 228, in assert_and_fix_args_for_test
suite_data = load_yaml(args.suite)
File "/home/azza/pythonenv/cwlair/lib/python3.8/site-packages/cwl_airflow/utilities/helpers.py", line 261, in load_yaml
raise ValueError
ValueError
Would you kindly advise?
Thank you much more. Azza
I also like to mention that the http://127.0.0.1:8080/admin/
page opens, with a 404 error message
Describe the bug I'm trying to run cwl-airflow on my laptop, following the instructions on the quick start guide.Then I tried installing airflow itself based on its quick local run instruction. Still, no success in installation.
Expected behavior cwl-airflow basic commands work fine
Screenshots and logs
Desktop (please complete the following information):
pip list
or similar command)