Installing extra Airflow packages causes errors when starting container

[tblazina]

I am on a Windows 10 machine, have pulled the image down and clone the repo. In the repo folder I run docker build --build-arg AIRFLOW_DEPS="gcp_api,google_auth,slack" -t puckel/docker-airflow . and then in docker-compose-LocalExecutor.yml I change the image name to puckel/docker-airflow:latest and run docker-compose -f docker-compose-LocalExecutor.yml up and there seem to be two problems.

1) I get the following error: env: bash\r: No such file or directory, which seems to be related to running this on Windows. I can resolve this by doing dos2unix entrypoint.sh when in the ./script folder, and then rebuilding the image.

2) After resolving that issue, when I start the container, I get the following error: ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package

When the image builds I see the following errors, which points to package version conflicts:

ERROR: flask 1.1.0 has requirement Jinja2>=2.10.1, but you'll have jinja2 2.10 which is incompatible.
ERROR: flask 1.1.0 has requirement Werkzeug>=0.15, but you'll have werkzeug 0.14.1 which is incompatible.
ERROR: google-cloud-bigquery 1.16.0 has requirement google-cloud-core<2.0dev,>=1.0.0, but you'll have google-cloud-core 0.28.1 which is incompatible.
ERROR: google-cloud-spanner 1.9.0 has requirement google-cloud-core<2.0dev,>=1.0.0, but you'll have google-cloud-core 0.28.1 which is incompatible.
ERROR: google-cloud-translate 1.5.0 has requirement google-cloud-core<2.0dev,>=1.0.0, but you'll have google-cloud-core 0.28.1 which is incompatible.

But I have not been able to figure out how to resolve this.

[bassco]

We are facing the flask 1.1.0 errors too, which was released 4 days ago. It's a side-effect of the upstream airflow 1.10.3 package not pinning the flask version. This change fixes the issue https://github.com/apache/airflow/pull/5535, but is unreleased for this project to make use of it.

[ttaschke]

With the current Airflow version used in this repository (whose flask version can not be changed here), it is necessary to workaround the problem by pinning the werkzeug version.

I prepared a PR here: https://github.com/puckel/docker-airflow/pull/393

[danielpoon]

Just FYI, I was doing a build just to add some addition Python modules and the container kept restarting complaining about werkzeug not found.

docker build --rm -t puckel/docker-airflow .

From reading the thread, that's when I realized you have removed it from setup.py. My fix was to add this back in. So I guess if you needed extra Airflow packages your fix is to remove it (which I have not tried). All I know is for me to build my own image I have to add it back. You may wish to reopen this for a more permanent fix

[vanducng]

This is due to the incompatible with current Flask's dependencies. The work around is simply execute the pip installation of Flask 1.0.4. pip install Flask==1.0.4

[rbk]

This is due to the incompatible with current Flask's dependencies. The work around is simply execute the pip installation of Flask 1.0.4. pip install Flask==1.0.4

Thank you! @vanducng

[RidleyLeisy]

If you're using conda, conda install -c anaconda flask==1.0.3 worked for me. Couldn't find flask==1.0.4 in anaconda distribution.

[m13m]

@vanducng Thanks :)

[thalesmello]

I am personally having problems with Flask==1.0.4 when using the UI

But Flask==1.0.3 seems to work just fine.

[zachliu]

I needed boto3 and had the same error:

ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package

Here is what I did in the Dockerfile:

...
ARG PYTHON_DEPS="boto3"
...
&& pip install 'Flask==1.0.4' \    # after the redis==3.2 line
...

[kochkina]

This is due to the incompatible with current Flask's dependencies. The work around is simply execute the pip installation of Flask 1.0.4. pip install Flask==1.0.4

it helped! Thank you a lot!

[pjroth]

I'm getting a similar error but I'm using 1.10.4rc3 since that was required to work around other dependency issues due to https://issues.apache.org/jira/browse/AIRFLOW-4900 so I used Flask==1.1.1 (the latest flask 1.X.Y version released as of today).

Specifying version ranges helps to create these problems by making builds non-reproducible. Lock your dependency versions to a specific version.

[zhangruiskyline]

We are facing the flask 1.1.0 errors too, which was released 4 days ago. It's a side-effect of the upstream airflow 1.10.3 package not pinning the flask version. This change fixes the issue apache/airflow#5535, but is unreleased for this project to make use of it.

Thanks, can I know whether the next release can have this merge? also if not, which is the version we should install to prevent this happen?

[pjroth]

Using 1.10.4 release I no longer need to specify a flask version to get airflow to startup (as of today at least) so it looks like the new version of airflow fixes this issue and this issue can be closed from my perspective.