Closed superbsky closed 2 years ago
Hello @superbsky,
Thank you for using CWL-Airflow. The log you provided indicates, that webserver and apiserver are waiting until cwl-airflow finishes creating all required databases (running airflow init). It shouldn't take a long time, however, if the setting in your .env
are not valid, that can be the reason why you can't start docker-compose. Let me know if it helps, or if you have any other questions.
I have fresh airflow install in $AIRFLOW_HOME and .env file updated accordingly. I waited for a few minutes, but init is not happening.
Also, I noticed that docker-compose causing some strange permission settings:
ls -la /temp/airflow_standalone/mysql_data
total 176200
drwxr-xr-x 6 999 root 4096 Jul 27 15:18 .
drwxrwxr-x 8 genx genx 4096 Jul 27 15:16 ..
drwxr-x--- 2 999 999 4096 Jul 27 15:17 airflow
-rw-r----- 1 999 999 56 Jul 27 15:16 auto.cnf
-rw------- 1 999 999 1680 Jul 27 15:16 ca-key.pem
-rw-r--r-- 1 999 999 1112 Jul 27 15:16 ca.pem
-rw-r--r-- 1 999 999 1112 Jul 27 15:16 client-cert.pem
-rw------- 1 999 999 1680 Jul 27 15:16 client-key.pem
-rw-r----- 1 999 999 667 Jul 27 15:18 ib_buffer_pool
-rw-r----- 1 999 999 50331648 Jul 27 15:18 ib_logfile0
-rw-r----- 1 999 999 50331648 Jul 27 15:16 ib_logfile1
-rw-r----- 1 999 999 79691776 Jul 27 15:18 ibdata1
drwxr-x--- 2 999 999 4096 Jul 27 15:16 mysql
drwxr-x--- 2 999 999 4096 Jul 27 15:16 performance_schema
-rw------- 1 999 999 1680 Jul 27 15:16 private_key.pem
-rw-r--r-- 1 999 999 452 Jul 27 15:16 public_key.pem
-rw-r--r-- 1 999 999 1112 Jul 27 15:16 server-cert.pem
-rw------- 1 999 999 1676 Jul 27 15:16 server-key.pem
drwxr-x--- 2 999 999 12288 Jul 27 15:17 sys
I just tried a fresh airflow installation using https://airflow.apache.org/docs/apache-airflow/stable/start/local.html, but w/o airflow db init
and it is still failing with missing table :(
May I missing something from the documentation? https://cwl-airflow.readthedocs.io/en/latest/readme/how_to_use.html#running-cwl-airflow-with-docker-compose
What are the exact steps to set up cwl-airflow w/ airflow as a standalone? as a docker? as a combination?
@michael-kotliar, any suggestions?
I am also getting some wired errors like:
scheduler | Traceback (most recent call last):
scheduler | File "/usr/local/bin/cwl-airflow", line 15, in <module>
scheduler | sys.exit(main(sys.argv[1:]))
scheduler | File "/usr/local/bin/cwl-airflow", line 9, in main
scheduler | args = parse_arguments(argsl)
scheduler | File "/usr/local/lib/python3.8/dist-packages/cwl_airflow/utilities/parser.py", line 281, in parse_arguments
scheduler | args, _ = get_parser().parse_known_args(argsl)
scheduler | File "/usr/local/lib/python3.8/dist-packages/cwl_airflow/utilities/parser.py", line 66, in get_parser
scheduler | version=get_version(),
scheduler | File "/usr/local/lib/python3.8/dist-packages/cwl_airflow/utilities/helpers.py", line 211, in get_version
scheduler | pkg = pkg_resources.require("cwl_airflow")
scheduler | File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 901, in require
scheduler | needed = self.resolve(parse_requirements(requirements))
scheduler | File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 792, in resolve
scheduler | raise VersionConflict(dist, req).with_context(dependent_req)
scheduler | pkg_resources.ContextualVersionConflict: (apache-airflow 2.0.1 (/usr/local/lib/python3.8/dist-packages), Requirement.parse('apache-airflow>=2.1.0'), {'apache-airflow-providers-http'})
By which I am very confused because I thought that cwl-airflow uses an installed version of the airflow...
What am I missing?
Hi @superbsky,
I've fixed bugs in the docker-compose file in the latest CWL-Airflow 1.2.11. Input parameters can be set in .env file. Additionally, you may take a look at the run_conformance_tests.sh file where I run our testing workflows using docker-compose.
Thanks for using CWL-Airflow
Feel free to reopen this issue if the problem still exists.
Describe the bug Running CWL-Airflow with docker-compose does not work w/
webserver | mysql: [Warning] Using a password on the command line interface can be insecure. webserver | ERROR 1146 (42S02) at line 1: Table 'airflow.dag_run' doesn't exist webserver | Sleep 1 sec apiserver | mysql: [Warning] Using a password on the command line interface can be insecure. apiserver | ERROR 1146 (42S02) at line 1: Table 'airflow.dag_run' doesn't exist apiserver | Sleep 1 sec
To Reproduce Steps to reproduce the behavior:
Expected behavior cwl-airflow docker-compose stack up and running with all services in check
Desktop (please complete the following information): Ubuntu 18
Additional context Full back to this solution https://morioh.com/p/3531d754cab7 but it also has some issues with job submission