No module named 'airflow.providers.google.common.hooks.leveldb'

[JustMaris]

Apache Airflow version: 2.0.2

Kubernetes version (if you are using kubernetes) (use kubectl version): v1.18.18

Environment: Cloud provider or hardware configuration: AWS

What happened:

Updated to Airflow 2.0.2 and a new warning appeared in webserver logs:

WARNING - Exception when importing 'airflow.providers.google.common.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common.hooks.leveldb'

What you expected to happen: No warning.

How to reproduce it: Don't know the specific details. Have tried adding pip install --upgrade apache-airflow-providers-google but the error was still there.

Anything else we need to know: I am not using LevelDB for anything in my code, as a result I don't understand from where this error is coming.

[ashb]

This I think can be ignored -- it is the webserver trying to load all the providers and it's failing on one, but this is not fatal and shouldn't stop the Webserver working.

Did 2.0.1 (or 2.0.0) have this same behaviour?

[JustMaris]

Airflow 2.0.1 and 2.0.0 did not have this behaviour. I've been actively checking logs recently with the upgrade to Airflow 2, so I can say that this warning was not there. I thought that this warning could be ignored, it's just coming up quite often in the logs, thus increasing their length.

[potiuk]

Thanks @ClassyLion.

It's a bug (harmless) in definition of the google provider 2.2.0 in fact:

In provider.yaml:

airflow.providers.google.common.hooks.leveldb.LevelDBHook

should be:

airflow.providers.google.leveldb.hooks.LevelDBHook

[KIRY4]

I'm not sure if it related with mentioned error. But now I'm trying to login to freshly installed airflow 2.0.2 with LDAP integration and I can't login with my credentials. Log of web pod full of mentioned error/warning:

WARNING - Exception when importing 'airflow.providers.google.common.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common.hooks.leveldb'.

airflow@airflow-web-749bb5988d-hv8xj:/opt/airflow$ airflow version
2.0.2

Everything works fine for me with 2.0.0 and 2.0.1 with my credentials from AD. My Airflow running on AKS if it matters.

Here is my webserver_config.py (maybe it also will be useful):

import os
from airflow import configuration as conf
from flask_appbuilder.security.manager import AUTH_LDAP
basedir = os.path.abspath(os.path.dirname(__file__))

SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')

CSRF_ENABLED = True

AUTH_TYPE = AUTH_LDAP

AUTH_ROLE_ADMIN = 'Admin'
AUTH_USER_REGISTRATION = True

#AUTH_USER_REGISTRATION_ROLE = "Admin"
AUTH_USER_REGISTRATION_ROLE = "User"

AUTH_LDAP_SERVER = 'ldaps://***.com:636'
AUTH_LDAP_SEARCH = "DC=***,DC=com"
AUTH_LDAP_BIND_USER = '***'
AUTH_LDAP_BIND_PASSWORD = '***'
AUTH_LDAP_UID_FIELD = '***'
AUTH_LDAP_USE_TLS = False
AUTH_LDAP_ALLOW_SELF_SIGNED = False
AUTH_LDAP_TLS_CACERTFILE = '/opt/airflow/ldap/ldap_ca.crt'

[potiuk]

@KIRY4 - can you see if you have error in your logs similar to:

ERROR - LDAP search for ........ returned multiple results

We think we know what the issue could be (related to Flask Application Builder https://github.com/dpgaspar/Flask-AppBuilder/issues/1581 and we might have a quick fix for that soon.

Can you also let us know how you installed Airflow ? Was this the airflow image ? Or another way ?

[KIRY4]

@potiuk

I'm building bit customized image using Breeze:

AIRFLOW_VERSION=2.0.2

./breeze-airflow/breeze build-image \
    --production-image  --python 3.8 --install-airflow-version=$AIRFLOW_VERSION \
    --additional-extras=doc,jdbc,odbc,oracle,snowflake --additional-python-deps="jaydebeapi azure-storage-blob" \
    --image-tag myacrdev01.azurecr.io/airflow:$AIRFLOW_VERSION-base

Later deploying it using following Helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow on AKS 1.18.14.

And yes, I'm getting following in logs from web pod: [2021-04-21 08:59:23,812] {manager.py:887} ERROR - LDAP search for '(sAMAccountName=***)' in scope 'DC=***,DC=com' returned multiple results

[potiuk]

For the future reference @KIRY4 - this is the root cause (added in FAB 3.2.0). For now i fixed constraints viaa #15470 but in a few days when FAB 3.2.3 gets released we will be able to us this: https://github.com/dpgaspar/Flask-AppBuilder/pull/1602#issuecomment-823962936

[potiuk]

You can rebuild your image with --force-pull-image flag and it should work fine now @KIRY4

[potiuk]

BTW. I recommend to use corresponding docker build . command to build the image rather than breeze.

See the latest documentation for this with nice examples: https://airflow.apache.org/docs/docker-stack/build.html#customizing-the-image

It's much simpler now to build the images and when you use docker build commands by default the released packages are used from PYPI to build the image rather than local source code. Unless you want to use latest master, version of Airflow/providers - this is the recommended way now.

[potiuk]

@KIRY4 -> would it be possible that you check if upgrading FAB to 3.2.3 ( released yesterday) fixes the problem?

[KIRY4]

@potiuk fixed! now everything works perfect! thank you! I mean that I successfully login but warning still exist in logs:

WARNING - Exception when importing 'airflow.providers.google.common.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common.hooks.leveldb'

[potiuk]

WARNING - Exception when importing 'airflow.providers.google.common.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common.hooks.leveldb'

Yep. This will fixed when we release next Google Provider and you update either the provider or, Airlfow (we plan to release 2.0.3 very soon addressing a small subset of changes + PIP 21 installability). PR here: https://github.com/apache/airflow/pull/15576

[maryampashmi]

airflow.providers.google.leveldb.hooks.LevelDBHook

This is not solved. I am getting the same error. Just tried to create a docker to install the module separately and this is the error I am getting

=> ERROR [2/2] RUN pip install --upgrade "pip==21.1.1" &&     pip install airflow.providers.google.leveldb.hooks.LevelDBHook &&     pip install   7.7s
------                                                                                                                                                  
 > [2/2] RUN pip install --upgrade "pip==21.1.1" &&     pip install airflow.providers.google.leveldb.hooks.LevelDBHook &&     pip install apache-airflow-providers-oracle==1.1.0 &&     pip install apache-airflow-providers-mysql==1.1.0  &&     pip install azure-storage-blob:                               
#5 0.265 + pip install --upgrade pip==21.1.1                                                                                                            
#5 0.990 Requirement already satisfied: pip==21.1.1 in /home/airflow/.local/lib/python3.6/site-packages (21.1.1)                                        
#5 6.744 + pip install airflow.providers.google.leveldb.hooks.LevelDBHook
#5 7.610 ERROR: Could not find a version that satisfies the requirement airflow.providers.google.leveldb.hooks.LevelDBHook (from versions: none)
#5 7.610 ERROR: No matching distribution found for airflow.providers.google.leveldb.hooks.LevelDBHook
------
executor failed running [/bin/bash -o pipefail -e -u -x -c pip install --upgrade "pip==21.1.1" &&     pip install airflow.providers.google.leveldb.hooks.LevelDBHook &&     pip install apache-airflow-providers-oracle==1.1.0 &&     pip install apache-airflow-providers-mysql==1.1.0  &&     pip install azure-storage-blob]: exit code: 1

What I suggest is to install complete package pip install apache-airflow-providers-google

[potiuk]

It is solved in the most recent version of the google provider (3.0.0). Currently when you install 2.0.2 you get the previous version of provider installed, but you can manually upgrade it - by extending the image and adding

pip install --upgrade apache-airflow-providers-google

It will be solved by out-of-the-box installation when we release 2.0.3 and 2.1

[maryampashmi]

Using --upgrade, solved the error. I haven't get it anymore. Thanks for your help.

[oliakhovych]

I have the same problem during completing command: docker-compose up Output: WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package: No module named 'plyvel' I tried this (didn't help me): pip install --upgrade apache-airflow-providers-google

[potiuk]

Upgrading dependencies in docker-compose requires you to update the image, not upgrade dependencie manually. You should read more how docker-compose works - see the note in https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.

If you want to get an easy to configure Docker-based deployment that Airflow Community develops, supports and can provide support with deployment, you should consider using Kubernetes and deploying Airflow using Official Airflow Community Helm Chart.

If you are not ready to manage and build your images, don't use docker compose other than just for quick start. Use local quick start instead: https://airflow.apache.org/docs/apache-airflow/stable/start/local.html