ImportError: cannot import name 'bytes_type' from 'oauthlib.common'

[ayush-san]

Apache Airflow version: 1.10.10 Environment: Airflow ECS

• Cloud provider or hardware configuration: Running on Docker
• OS (e.g. from /etc/os-release): AWS Linux 2
• Kernel (e.g. uname -a): Linux
• Install tools:
• Others:

I have made some change in Dockerfile and added these lines

ARG WEBSERVER_CONFIG="default_webserver_config.py"
ENV WEBSERVER_CONFIG=${WEBSERVER_CONFIG}

COPY --chown=airflow:airflow "./airflow/config_templates/${WEBSERVER_CONFIG}" ${AIRFLOW_HOME}/webserver_config.py

ARG AIRFLOW_CFG="default_airflow.cfg"
ENV AIRFLOW_CFG=${AIRFLOW_CFG}

COPY --chown=airflow:airflow "./airflow/config_templates/${AIRFLOW_CFG}" ${AIRFLOW_HOME}/airflow.cfg

What happened I am updating airflow from 1.10.5 to 1.10.10. I am getting the following error in webserver when using google OAuth.

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
    app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 295, in cached_app
    app, _ = create_app(config, session, testing)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 108, in create_app
    update_perms=conf.getboolean('webserver', 'UPDATE_FAB_PERMS'))
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/base.py", line 148, in __init__
    self.init_app(app, session)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/base.py", line 202, in init_app
    self.sm = self.security_manager_class(self)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 177, in __init__
    super(AirflowSecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/sqla/manager.py", line 51, in __init__
    super(SecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/manager.py", line 249, in __init__
    from flask_oauthlib.client import OAuth
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_oauthlib/client.py", line 20, in <module>
    from .utils import to_bytes
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_oauthlib/utils.py", line 5, in <module>
    from oauthlib.common import to_unicode, bytes_type
ImportError: cannot import name 'bytes_type' from 'oauthlib.common' (/home/airflow/.local/lib/python3.7/site-packages/oauthlib/common.py)

This error is because of the dependency mismatch. Airflow 1.10.10 is installing oauthlib==3.1.0 and Flask-OAuthlib==0.9.5 and in 3.1.0 version of oauthlib, there is no bytes_type while Flash-OAuthlib is expecting it to be there.

Considering that lepture has archived flask-oauthlib and created authlib, I see a PR in airflow from lepture about this change https://github.com/apache/airflow/pull/6140 but it is not merged and marked stale by the bot

Has anyone faced this issue in Airflow 1.10.10? I have fixed this issue by changing the oauthlib version to 2.1.0 in requirement.txt

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[bensta]

I face the same issue, forcing oauthlib 2.1.0 does not solve the issue for me. I am testing on Minikube, official apache/airflow-python3.6 image, trying to run with flask RBAC,

[ayush-san]

@feng-tao Can you please look into this issue as this is preventing us from upgrading airflow

[waldher]

It looks like this was introduced here: https://github.com/apache/airflow/blame/96697180d79bfc90f6964a8e99f9dd441789177c/requirements/requirements-python3.5.txt#L255

This is because Flask-Oauthlib is only compatible with requests-oauthlib==1.1.0, and this update pinned it at 1.3.0

[ayush-san]

@waldher Yes and lepture has archived flask-oauthlib and created authlib, I see a PR in airflow from lepture about this change https://github.com/apache/airflow/pull/6140 but it is not merged and marked stale by the bot

Also is anyone working on fixing this issue?

[andreyvital]

Also struggling with this. Right now we're using Pomerium as a proxy in front of our internal Airflow: https://github.com/pomerium/pomerium

[joshzana]

Any update on this bug? It appears to only affect usage of airflow within the official docker image, as the unofficial puckel/docker image does not hit the bug with version 1.10.10 and the same Airflow RBAC + Google Auth configuration.

For those stuck on this, here's a workaround that worked for me. Make a custom dockerfile and override the versions of the two libraries:

FROM apache/airflow:1.10.12-python3.8
RUN pip install --user -I requests_oauthlib==1.1.0 \
    && pip install --user  -I oauthlib==2.1.0

YMMV

[ayush-san]

@joshzana They have made the required changes for Airflow 2.0 https://github.com/apache/airflow/blob/master/UPDATING.md#breaking-change-in-oauth.

[zachliu]

@joshzana I'm also using puckel/docker image with Airflow 1.10.12 (RBAC + Google Auth configuration). I only see this error if I use the constraints-1.10.12/constraints-3.8 file. If this constraints file is not compatible with 1.10.12, can we just replace it with a correct one instead?

I just started to use the constraint files because of https://github.com/apache/airflow/issues/11965

[joshzana]

@zachliu - not sure about that, but it looks like #11965 has been resolved by a hotfix to cattrs, so you may want to take a look at that.

[zachliu]

@joshzana that fix doesn't apply to me, i'm using python 3.8

[weldpua2008]

@joshzana still reproduces on 1.10.14 and Python 3.6

$ airflow webserver
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 81, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1179, in webserver
    app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
  File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 297, in cached_app
    app, _ = create_app(config, session, testing)
  File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 115, in create_app
    update_perms=conf.getboolean('webserver', 'UPDATE_FAB_PERMS'))
  File "/usr/local/lib/python3.6/site-packages/flask_appbuilder/base.py", line 148, in __init__
    self.init_app(app, session)
  File "/usr/local/lib/python3.6/site-packages/flask_appbuilder/base.py", line 202, in init_app
    self.sm = self.security_manager_class(self)
  File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/security.py", line 180, in __init__
    super(AirflowSecurityManager, self).__init__(appbuilder)
  File "/usr/local/lib/python3.6/site-packages/flask_appbuilder/security/sqla/manager.py", line 51, in __init__
    super(SecurityManager, self).__init__(appbuilder)
  File "/usr/local/lib/python3.6/site-packages/flask_appbuilder/security/manager.py", line 249, in __init__
    from flask_oauthlib.client import OAuth
  File "/usr/local/lib/python3.6/site-packages/flask_oauthlib/client.py", line 20, in <module>
    from .utils import to_bytes
  File "/usr/local/lib/python3.6/site-packages/flask_oauthlib/utils.py", line 5, in <module>
    from oauthlib.common import to_unicode, bytes_type
ImportError: cannot import name 'bytes_type'

[potiuk]

You probably have wrong version of flask_appbuilder. Please use constraints to upgrade your version of airflow as described here http://airflow.apache.org/docs/apache-airflow/stable/installation.html (using constraint files). You can run pip freeze | sort in your environment and compare your versions with https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt

[weldpua2008]

You probably have wrong version of flask_appbuilder. Please use constraints to upgrade your version of airflow as described here http://airflow.apache.org/docs/apache-airflow/stable/installation.html (using constraint files). You can run pip freeze | sort in your environment and compare your versions with https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt

Hey @potiuk , We are using constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt for installation

[potiuk]

| We are using constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt for installation

Can you please verify that you have the same version of dependencies as in the constraints? What the comparison shows when you run diff vs. pip freeze | sort ?

[dkinzer]

@potiuk, we are seeing the same issue as @weldpua2008 using airflow 1.10.14 and applying the 1.10.14/constraints-3.6.txt.

I've determined that the constraints do not specify a version for flask_oauthlib even though they make a direct dependency via the flask_appbuilder package and this is likely at the root of the problem:

Flask-AppBuilder==2.3.4 has a dep to flask_oauthlib but the constraint do not specify this package https://github.com/dpgaspar/Flask-AppBuilder/blob/v2.3.4/flask_appbuilder/security/manager.py#L249

I've deleted my local copy of flask_oauthlib and tried to reinstall it using the constraint and I get the following error:

pip install flask_oauthlib --constraint=https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt
flask-oauthlib 0.9.6 has requirement oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2, but you'll have oauthlib 3.1.0 which is incompatible.
flask-oauthlib 0.9.6 has requirement requests-oauthlib<1.2.0,>=0.6.2, but you'll have requests-oauthlib 1.3.0 which is incompatible.

So I think this incompatibility is at the heart of the issue.

[dkinzer]

I'm able to get past issue using this modified constraint ☝🏽 which updates the 1-10 constraints with some changes added to 2-0 constraints.

[namiops]

We faced same issue and temporary fix it by force upgrade flask_oauthlib from 0.9.5 to 0.9.6: pip install flask_oauthlib==0.9.6

[snigdhasaikam30]

I'm able to get past issue using this modified constraint ☝🏽 which updates the 1-10 constraints with some changes added to 2-0 constraints.

Hey @dkinzer I've tried setting flask_oauthlib to 0.9.5, requests-oauthlib to 1.1.0 and oauthlib to 2.1.0 but that didn't work for me. I'm using airflow version 1.10.14 and python 3.6

[snigdhasaikam30]

We faced same issue and temporary fix it by force upgrade flask_oauthlib from 0.9.5 to 0.9.6: pip install flask_oauthlib==0.9.6

hey @namiops what version of airflow are you using? and python version?

[dkinzer]

@snigdhasaikam30 try using --constraint=https://raw.githubusercontent.com/apache/airflow/cc382b75783d76888fcd5b0e38228f08ee83dbd5/constraints-3.6.txt when running the pip install command.

That constraint is the one that is working for me that I have in #14562. I'm using the same python version and airflow version that you are.

[snigdhasaikam30]

@snigdhasaikam30 try using --constraint=https://raw.githubusercontent.com/apache/airflow/cc382b75783d76888fcd5b0e38228f08ee83dbd5/constraints-3.6.txt when running the pip install command.

That constraint is the one that is working for me that I have in #14562. I'm using the same python version and airflow version that you are.

This is what worked for us. changed the user to "airflow" instead of root

USER airflow RUN pip install --user -I flask_oauthlib==0.9.6 --constraint=https://raw.githubusercontent.com/apache/airflow/cc382b75783d76888fcd5b0e38228f08ee83dbd5/constraints-3.6.txt

airflow:1.10.15-python3.6

[ashb]

Looks like this has been fixed, or at least worked around now on 1.10, and is gone entirely on 2.0.

[namiops]

hey @namiops what version of airflow are you using? and python version?

hi @snigdhasaikam30, I'm using Airfllow 1.10.15 with python 3.8.