ModuleNotFoundError installing Superset 2.0.1 in a fresh virtual environment

[alanorth]

Trying to install Superset 2.0.1 in a fresh Python 3.8.10 virtual environment fails with ModuleNotFoundError.

How to reproduce the bug

# su - superset
$ python3.8 -m venv venv
$ source ./venv/bin/activate
$ pip install --upgrade setuptools pip wheel
$ pip install apache-superset==2.0.1
$ export PYTHONPATH=/home/superset
$ export FLASK_APP=superset
$ superset
Traceback (most recent call last):
  File "/home/superset/venv/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/home/superset/venv/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/superset/venv/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/superset/venv/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/superset/venv/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/superset/venv/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/superset/venv/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

But clearly cryptography is installed:

$ pip install cryptography
Requirement already satisfied: cryptography in ./venv/lib/python3.8/site-packages (39.0.0)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.8/site-packages (from cryptography) (1.15.1)
Requirement already satisfied: pycparser in ./venv/lib/python3.8/site-packages (from cffi>=1.12->cryptography) (2.21)

Expected results

Superset runs.

Actual results

ModuleNotFoundError from Python.

Environment

System is Ubuntu 20.04 "focal" with Python 3.8.10.

Checklist

• [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
• [x] I have reproduced the issue with at least the latest released version of superset.
• [x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

The system is currently running Superset 2.0.0 fine in another virtualenv.

[alanorth]

I just noticed that Superset 2.0.1's requirements/base.txt says it wants cryptography==3.4.7. If I install that manually then superset runs as expected.

I have different ModuleNotFoundErrors every time I install a new version of Superset (I always file an issue here). In fact, looking at pip list in my virtual environment, I see several other packages that are the wrong version. What is going on with the Python packaging of the pip version that causes pip install apache-superset==2.0.1 to not install the exact version dependencies? Why do the Superset docs still recommend using the pip install method at all?

[io-ma]

I am having the exact same issue: ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

Python 3.8.16, Ubuntu 18.04 LTS

superset db upgrade
Traceback (most recent call last):
  File "/home/io/superset/venv/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset==2.0.1', 'console_scripts', 'superset')())
  File "/home/io/superset/venv/bin/superset", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

[zhaoyongjie]

@alanorth try this snippet

pip uninstall cryptography
pip uninstall pyopenssl
pip install cryptography==3.4.7

[alanorth]

@zhaoyongjie I ended up using this one liner shared in another issue. It installs the exact dependencies for Apache Superset 2.0.1, from its requirements/base.txt:

$ curl -sS https://raw.githubusercontent.com/apache/superset/2.0.1/requirements/base.txt | \
       tail -n +10 | \
       awk -v ORS=" " '/^[A-z]/{print}' | \
       xargs pip install apache-superset==2.0.1

I still don't understand why pip installing apache-superset==2.0.1 doesn't work. This is a recurring issue with Superset for me.

[io-ma]

this one liner worked for me too, thank you!

[Jika97]

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

[zhaoyongjie]

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

Are you on the Superset 2.0.1? I can't reproduce it on the master and 2.0.1.

[Jika97]

Yes, I'm trying install Superset 2.0.1 on Rocky Linux 9.1 with Python 3.9 @zhaoyongjie, but i will try on Ubuntu 20.04

[xinbeiG]

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

I got the same issue here, and this solution fixed it.

pip uninstall WTForms 
pip install WTForms==2.3.0

[Jika97]

Thanks @XinbeiGong, it's worked !

I think it would be necessary to update the installation documentation, because it is a bit tedious to search all over the internet to install and operate this magnificent tool.

Indirectly this repels a lot of potential users.

[michael-s-molina]

Tagging @eschutho @eric-briscoe @AAfghahi as owners of 2.0.1

[jsanko9]

Same problem Apply the solution of @zhaoyongjie, now i have : from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

I got the same issue here, and this solution fixed it.

pip uninstall WTForms 
pip install WTForms==2.3.0

I was having this issue plus an issue with validate function: validate() takes 1 positional argument but 2 were given

I think culprit was with WTForms, updating to 2.3.2 seems to have helped, looking at the change log they may have changed something in 2.3.1 https://wtforms.readthedocs.io/en/2.3.x/changes/#version-2-3-1 Posting it here in case someone encounters same thing.

pip uninstall WTForms 
pip install WTForms==2.3.2

[eschutho]

Thanks for that info @jsanko9. It looks like we can pin WTForms to >=2.3.2 <3.0 and that should help with that issue. It looks like there may be a similar issue with the cryptography package. We know that cryptography==3.4.7 works. We currently require "cryptography>=3.3.2". Does anyone know which version is failing? cc @zhaoyongjie?

[noahyetter-nb]

Thanks for that info @jsanko9. It looks like we can pin WTForms to >=2.3.2 <3.0 and that should help with that issue. It looks like there may be a similar issue with the cryptography package. We know that cryptography==3.4.7 works. We currently require "cryptography>=3.3.2". Does anyone know which version is failing? cc @zhaoyongjie?

I just did a fresh install and got cryptography 39.0.0 See also this issue https://github.com/apache/superset/issues/22743 I applied the fix reported there, which works, but now I have the problem with wtforms described above

(I can't help but notice that the dependencies that get installed by the process in Dockerfile, and the dependencies specified in setup.py, are significantly different. Also the non-Docker installation instructions are incorrect.)

[alonsogtouch]

I got the same error on Fedora 37.

Just so you know I couldn't get it to work with Python default version on Fedora (3.11) since it's not compatible with lots of libraries and dependencies in Superset.

In case anyone is interested I first had to build Python 3.8 from source:

sudo dnf groupinstall "Development Tools"
sudo dnf install python3-devel openssl-devel zlib-devel bzip2-devel sqlite-devel libffi-devel
wget https://www.python.org/ftp/python/3.8.16/Python-3.8.16.tgz
tar xzf Python-3.8.16.tgz Python-3.8.16/
cd Python-3.8.16/
./configure --enable-optimizations --with-ensurepip=install --with-lto=full --enable-profiling
sudo make altinstall

This last line helps you leave your system-wide Python 3.11 intact. You can run python3.8 after that and create a superset virtual enrivonment using

python3.8 -m venv superset

After activating the new virtual environment and installing apache-superset with pip I got errors when I ran

superset db upgrade

as stated in the documentation (https://superset.apache.org/docs/installation/installing-superset-from-scratch/):

(superset) [alonso@fedora pythonvenvs]$ superset db upgrade
/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/pandas/compat/__init__.py:124: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/home/alonso/pythonvenvs/superset/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

In order to fix this I had to run the lines @zhaoyongjie provided, which I reproduce below:

@alanorth try this snippet

pip uninstall cryptography
pip uninstall pyopenssl
pip install cryptography==3.4.7

After that I ran

export FLASK_APP=superset
superset db upgrade

And I got the following errors:

  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/initialization/__init__.py", line 476, in configure_data_sources
    ConnectorRegistry.register_sources(module_datasource_map)
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/connector_registry.py", line 42, in register_sources
    module_obj = __import__(module_name, fromlist=class_names)
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/sqla/__init__.py", line 17, in <module>
    from . import models, views
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/sqla/views.py", line 28, in <module>
    from wtforms.ext.sqlalchemy.fields import QuerySelectField
ModuleNotFoundError: No module named 'wtforms.ext'

@jsanko9 helped with this, I'm partly quoting the reply:

pip uninstall WTForms 
pip install WTForms==2.3.2

With that I could successfully run

superset db upgrade

[zeynepkoyun]

@alanorth Hi, You can try with pip install pyopenssl==22.1.0. Downloading this library fixed it for me.

[cwegener]

FWIW, there is a merged PR that removes the use of the private cryptography API _Certificate class by replacing it with the public version (Certificate) and at the same time pins the cryptography version.

https://github.com/apache/superset/pull/22744

[prakharcode]

Fixed installation, working on ubuntu 22.10

pyenv version

3.9.16 (set by /root/.pyenv/version)

pip freeze

aiohttp==3.8.4
aiosignal==1.3.1
alembic==1.9.4
amqp==5.1.1
apache-superset==2.0.1
apispec==5.2.2
async-generator==1.10
async-timeout==4.0.2
attrs==22.2.0
Babel==2.11.0
backoff==2.2.1
billiard==3.6.4.0
bleach==3.3.1
Brotli==1.0.9
cachelib==0.4.1
celery==5.2.7
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.0.1
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.6
convertdate==2.4.0
cron-descriptor==1.2.35
croniter==1.3.8
cryptography==38.0.2
deprecation==2.1.0
dnspython==2.3.0
email-validator==1.3.1
exceptiongroup==1.1.0
Flask==2.1.0
Flask-AppBuilder==4.2.0
Flask-Babel==2.0.0
Flask-Caching==1.11.1
Flask-Compress==1.13
Flask-JWT-Extended==4.4.4
Flask-Login==0.6.2
Flask-Migrate==4.0.4
Flask-SQLAlchemy==2.5.1
flask-talisman==1.0.0
Flask-WTF==1.1.1
frozenlist==1.3.3
func-timeout==4.3.5
geographiclib==2.0
geopy==2.3.0
graphlib-backport==1.0.3
gunicorn==20.1.0
h11==0.14.0
hashids==1.3.1
hijri-converter==2.2.4
holidays==0.14.2
humanize==4.6.0
idna==3.4
importlib-metadata==6.0.0
isodate==0.6.1
itsdangerous==2.1.2
Jinja2==3.1.2
jsonschema==4.17.3
kombu==5.2.4
korean-lunar-calendar==0.3.1
Mako==1.2.4
Markdown==3.4.1
MarkupSafe==2.0.1
marshmallow==3.19.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.26.1
msgpack==1.0.4
multidict==6.0.4
numpy==1.22.1
outcome==1.2.0
packaging==23.0
pandas==1.3.5
parsedatetime==2.6
pgsanity==0.2.9
polyline==2.0.0
prison==0.2.1
prompt-toolkit==3.0.36
psycopg2-binary==2.9.5
pyarrow==5.0.0
pycparser==2.21
PyJWT==2.6.0
PyMeeus==0.5.12
pyparsing==3.0.9
pyrsistent==0.19.3
PySocks==1.7.1
python-dateutil==2.8.2
python-dotenv==0.21.1
python-geohash==0.8.5
pytz==2022.7.1
PyYAML==6.0
redis==4.5.1
selenium==4.8.0
simplejson==3.18.3
six==1.16.0
slackclient==2.5.0
sniffio==1.3.0
sortedcontainers==2.4.0
SQLAlchemy==1.3.24
SQLAlchemy-Utils==0.37.9
sqlparse==0.3.0
tabulate==0.8.9
trio==0.22.0
trio-websocket==0.9.2
typing-extensions==3.10.0.2
urllib3==1.26.14
vine==5.0.0
wcwidth==0.2.6
webencodings==0.5.1
Werkzeug==2.0.0
wsproto==1.2.0
WTForms==2.3.3
WTForms-JSON==0.3.5
yarl==1.8.2
zipp==3.13.0

[EMCP]

Can confirm, Ubuntu 20.04 py3.8 got stood up successfully with the following

pip install --upgrade pip
sudo -u ubuntu pip install --upgrade pip --no-warn-script-location
sudo -u ubuntu pip install testresources --no-warn-script-location
sudo -u ubuntu pip install aiohttp==3.8.1 aiosignal==1.2.0 alembic==1.6.5 amqp==5.1.0 apispec[yaml]==3.3.2 async-timeout==4.0.2 attrs==21.2.0 babel==2.9.1 backoff==1.11.1 billiard==3.6.4.0 bleach==3.3.1 brotli==1.0.9 cachelib==0.4.1 celery==5.2.2 cffi==1.14.6 charset-normalizer==2.0.4 click==8.0.4 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 colorama==0.4.4 convertdate==2.4.0 cron-descriptor==1.2.24 croniter==1.0.15 cryptography==3.4.7 deprecation==2.1.0 dnspython==2.1.0 email-validator==1.1.3 flask==2.0.3 flask-appbuilder==4.1.3 flask-babel==1.0.0 flask-caching==1.10.1 flask-compress==1.10.1 flask-jwt-extended==4.3.1 flask-login==0.4.1 flask-migrate==3.1.0 flask-sqlalchemy==2.5.1 flask-talisman==0.8.1 flask-wtf==0.14.3 frozenlist==1.3.0 func-timeout==4.3.5 geographiclib==1.52 geopy==2.2.0 graphlib-backport==1.0.3 gunicorn==20.1.0 hashids==1.3.1 holidays==0.14.2 humanize==3.11.0 idna==3.2 isodate==0.6.0 itsdangerous==2.1.1 jinja2==3.0.3 jsonschema==3.2.0 kombu==5.2.4 korean-lunar-calendar==0.2.1 mako==1.1.4 markdown==3.3.4 markupsafe==2.0.1 marshmallow==3.13.0 marshmallow-enum==1.5.1 marshmallow-sqlalchemy==0.23.1 msgpack==1.0.2 multidict==5.1.0 numpy==1.22.1 packaging==21.3 pandas==1.3.4 parsedatetime==2.6 pgsanity==0.2.9 polyline==1.4.0 prison==0.2.1 prompt-toolkit==3.0.28 pyarrow==5.0.0 pycparser==2.20 pyjwt==2.4.0 pymeeus==0.5.11 pyparsing==3.0.6 pyrsistent==0.16.1 python-dateutil==2.8.2 python-dotenv==0.19.0 python-editor==1.0.4 python-geohash==0.8.5 pytz==2021.3 pyyaml==5.4.1 redis==3.5.3 selenium==3.141.0 simplejson==3.17.3 six==1.16.0 slackclient==2.5.0 sqlalchemy==1.3.24 sqlalchemy-utils==0.37.8 sqlparse==0.3.0 tabulate==0.8.9 typing-extensions==3.10.0.0 urllib3==1.26.6 vine==5.0.0 wcwidth==0.2.5 webencodings==0.5.1 werkzeug==2.0.3 wtforms==2.3.3 wtforms-json==0.3.3 yarl==1.6.3 --no-warn-script-location
sudo -u ubuntu pip install psycopg2-binary pillow gunicorn gevent --no-warn-script-location
sudo -u ubuntu pip install requests --upgrade --no-warn-script-location
sudo -u ubuntu pip install trino thrift pyhive --no-warn-script-location

status-set maintenance "About to pip install apache-superset $(date +"%H:%M")"
juju-log -l "WARNING" "About to pip install apache-superset"
sudo -u ubuntu pip install apache-superset --no-warn-script-location

[TrabajadorPraOleo]

I install specifics the versions for the frameworks (WTForms, cryptography, pyopenssl ...) but it has not been solved. I have checked the versions in the file requirements/base.txt.

The error comes out when I enter this command: -sudo docker-compose up

[CharlesYe8848]

您好，你发的邮件我已收到。

[alanorth]

Closing because we realized that pip install apache-superset is apparently not meant to work. In other threads the maintainers have commented that we're supposed to be using the requirements file with pip. :shrug:

For better or worse, now I always install Superset using a one liner based on the example given earlier in this issue:

$ curl -sS https://raw.githubusercontent.com/apache/superset/2.0.1/requirements/base.txt | \
       tail -n +10 | \
       awk -v ORS=" " '/^[A-z]/{print}' | \
       xargs pip install apache-superset==2.0.1