Airflow 2.0 - ModuleNotFoundError: No module named 'airflow.providers.http'

colinglaes commented 3 years ago

Apache Airflow version: 2.0.0

Environment:

Cloud provider or hardware configuration:
CPU: AMD Ryzen Threadripper 1950x
RAM: 64GB RAM
Motherboard: X399 Aorus Pro
OS (e.g. from /etc/os-release): 18.04.5 LTS (Bionic Beaver)
Kernel (e.g. uname -a): Linux orion 5.4.0-60-generic #67~18.04.1-Ubuntu SMP Tue Jan 5 22:01:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Install tools: https://airflow.apache.org/docs/apache-airflow/stable/installation.html & https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html
Python: 3.6.9

What happened:

I got to upgrading to airflow 2 but when i received the following error message when running airflow db upgrade

[cglaes@orion : ~ : 01:55] $ airflow db upgrade
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324: DeprecationWarning: The log_format option in [core] has been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
  option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324: DeprecationWarning: The simple_log_format option in [core] has been moved to the simple_log_format option in [logging] - the old setting has been used, but please update your config.
  option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The logging_config_class option in [core] has been moved to the logging_config_class option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The fab_logging_level option in [core] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_log_format option in [core] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_console_log option in [core] has been moved to the colored_console_log option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_formatter_class option in [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The dag_processor_manager_log_location option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The log_filename_template option in [core] has been moved to the log_filename_template option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The log_processor_filename_template option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The remote_logging option in [core] has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The task_log_reader option in [core] has been moved to the task_log_reader option in [logging] - the old setting has been used, but please update your config.
/home/cglaes/.local/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
DB: postgresql://localhost:5432/airflow
[2021-01-11 01:56:07,048] {db.py:678} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/home/cglaes/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
    db.upgradedb()
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 688, in upgradedb
    command.upgrade(config, 'heads')
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 511, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/command.py", line 287, in upgrade
    return script._upgrade_revs(revision, rev)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/base.py", line 364, in _upgrade_revs
    revs = list(revs)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 765, in _iterate_revisions
    requested_lowers = self.get_revisions(lower)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 319, in get_revisions
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 319, in <listcomp>
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 321, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 501, in _resolve_revision_number
    self._revision_map
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/util/langhelpers.py", line 230, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/revision.py", line 123, in _revision_map
    for revision in self._generator():
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/base.py", line 112, in _load_revisions
    script = Script._from_filename(self, vers, file_)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/script/base.py", line 906, in _from_filename
    module = util.load_python_file(dir_, filename)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/cglaes/.local/lib/python3.6/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 27, in <module>
    from airflow.www.app import create_app
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/www/app.py", line 38, in <module>
    from airflow.www.extensions.init_views import (
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
    from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/www/views.py", line 2836, in <module>
    class ConnectionFormWidget(FormWidget):
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/www/views.py", line 2839, in ConnectionFormWidget
    field_behaviours = json.dumps(ProvidersManager().field_behaviours)
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/providers_manager.py", line 397, in field_behaviours
    self.initialize_providers_manager()
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/providers_manager.py", line 129, in initialize_providers_manager
    self._discover_all_providers_from_packages()
  File "/home/cglaes/.local/lib/python3.6/site-packages/airflow/providers_manager.py", line 148, in _discover_all_providers_from_packages
    provider_info = entry_point.load()()
  File "/home/cglaes/.local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'airflow.providers.http'
[cglaes@orion : ~ : 01:56] $

What you expected to happen:

Successful upgrade with out

How to reproduce it:

Follow tutorial: https://airflow.apache.org/docs/apache-airflow/stable/installation.html & https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html

Anything else we need to know:

[cglaes@orion : ~ : 02:19] $ pip freeze
alembic==1.4.3
apache-airflow==2.0.0
apache-airflow-providers-ftp==1.0.0
apache-airflow-providers-http==1.0.0
apache-airflow-providers-imap==1.0.0
apache-airflow-providers-postgres==1.0.0
apache-airflow-providers-sqlite==1.0.0
apache-airflow-upgrade-check==1.1.0
apispec==3.3.2
apturl==0.5.2
argcomplete==1.12.2
asn1crypto==0.24.0
attrs==20.3.0
awscli==1.18.69
Babel==2.9.0
backcall==0.1.0
bitarray==1.2.1
bleach==2.1.2
botocore==1.16.19
bottle==0.12.13
Brlapi==0.6.6
cached-property==1.5.2
cachetools==4.2.0
cattrs==1.0.0
certifi==2020.11.8
cffi==1.14.4
chardet==3.0.4
click==6.7
clickclick==20.10.2
cloudpickle==1.3.0
colorama==0.4.4
colorlog==4.0.2
command-not-found==0.3
commonmark==0.9.1
configparser==3.5.3
connexion==2.7.0
croniter==0.3.36
cryptography==3.2.1
cupshelpers==1.0
cycler==0.10.0
dask==2.15.0
dataclasses==0.7
decorator==4.1.2
defer==1.0.6
defusedxml==0.6.0
dill==0.3.3
distributed==2.15.2
distro-info===0.18ubuntu0.18.04.1
dnspython==1.16.0
docker==4.2.0
docker-pycreds==0.2.1
docutils==0.15.2
email-validator==1.1.2
entrypoints==0.2.3.post1
flake8==3.7.9
Flask==1.1.2
Flask-Admin==1.5.4
Flask-AppBuilder==3.1.1
Flask-Babel==1.0.0
Flask-Caching==1.9.0
Flask-JWT-Extended==3.25.0
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.4
flask-swagger==0.2.13
Flask-WTF==0.14.3
fsspec==0.7.3
funcsigs==1.0.2
future==0.18.2
fxcmpy==1.2.6
Glances==2.11.1
google-auth==1.24.0
graphviz==0.15
gunicorn==19.10.0
HeapDict==1.0.1
html5lib==0.999999999
httplib2==0.9.2
idna==2.8
importlib-metadata==1.7.0
importlib-resources==1.5.0
impyla==0.16.2
inflection==0.5.1
influxdb==4.1.1
iotop==0.6
ipykernel==4.8.2
ipython==7.13.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
iso8601==0.1.13
itsdangerous==1.1.0
jedi==0.17.0
Jinja2==2.11.2
jmespath==0.9.3
json-merge-patch==0.2
json5==0.9.5
jsonschema==3.2.0
jupyter-client==5.2.2
jupyter-core==4.4.0
jupyterlab==2.2.4
jupyterlab-server==1.2.0
keyring==10.6.0
keyrings.alt==3.0
kubernetes==11.0.0
language-selector==0.1
launchpadlib==1.10.6
lazr.restfulclient==0.13.5
lazr.uri==1.0.3
lazy-object-proxy==1.4.3
locket==0.2.0
lockfile==0.12.2
louis==3.5.0
macaroonbakery==1.1.3
Mako==1.1.3
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==3.10.0
marshmallow-enum==1.5.1
marshmallow-oneofschema==2.0.1
marshmallow-sqlalchemy==0.23.1
matplotlib==2.1.1
mccabe==0.6.1
mistune==0.8.3
msgpack==1.0.0
mypy-extensions==0.4.3
natsort==7.1.0
nbconvert==5.3.1
nbformat==4.4.0
netifaces==0.10.4
notebook==5.2.2
numpy==1.19.4
oauth==1.0.1
oauthlib==3.1.0
olefile==0.45.1
openapi-spec-validator==0.2.9
packaging==20.7
pandas==1.1.4
pandocfilters==1.4.2
parso==0.7.0
partd==1.1.0
pendulum==2.1.2
pep562==1.0
pexpect==4.2.1
pickleshare==0.7.5
Pillow==5.1.0
ply==3.11
prison==0.1.3
prompt-toolkit==3.0.5
protobuf==3.0.0
psutil==5.7.3
psycopg2==2.8.5
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycodestyle==2.5.0
pycparser==2.20
pycrypto==2.6.1
pycryptodomex==3.4.7
pycups==1.9.73
pyflakes==2.1.1
Pygments==2.7.2
pygobject==3.26.1
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.1.2
pyparsing==2.4.7
pyRFC3339==1.0
pyrsistent==0.17.3
pysmi==0.2.2
pysnmp==4.4.3
pystache==0.5.4
python-apt==1.6.5+ubuntu0.5
python-box==4.2.3
python-daemon==2.2.4
python-dateutil==2.8.1
python-debian==0.1.32
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2020.4
pytzdata==2020.1
pyxdg==0.25
PyYAML==5.3.1
pyzmq==16.0.2
reportlab==3.4.0
requests==2.23.0
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
rich==9.2.0
roman==2.0.0
rsa==3.4.2
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
s3transfer==0.3.3
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
setproctitle==1.2
simplegeneric==0.8.1
simplejson==3.13.2
six==1.15.0
socketIO-client==0.7.2
sortedcontainers==2.1.0
SQLAlchemy==1.3.20
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.36.8
ssh-import-id==5.7
supervisor==4.1.0
swagger-ui-bundle==0.0.8
system-service==0.3
systemd-python==234
tabulate==0.8.7
tblib==1.6.0
tenacity==6.2.0
termcolor==1.1.0
terminado==0.7
testpath==0.3.1
text-unidecode==1.3
thrift==0.13.0
thriftpy2==0.4.11
toml==0.10.0
toolz==0.10.0
tornado==6.0.4
tqdm==4.42.1
traitlets==4.3.2
typing==3.7.4.3
typing-extensions==3.7.4.3
tzlocal==1.5.1
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unicodecsv==0.14.1
urllib3==1.25.11
usb-creator==0.3.3
virtualenv==16.7.4
wadllib==1.3.2
wcwidth==0.1.9
webencodings==0.5
websocket-client==0.44.0
Werkzeug==1.0.1
WTForms==2.3.3
xkit==0.0.0
zict==2.0.0
zipp==3.4.0
zope.deprecation==4.4.0
zope.interface==4.3.2
[cglaes@orion : ~ : 02:22] $

boring-cyborg[bot] commented 3 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!

potiuk commented 3 years ago

You probably have Airflow installed in --editable mode from sources rather than packages.

You should remove the 'apache-airflow-providers*' if you install airflow in --editable mode (Though I would argue this is not the best Idea to run airflow in --editable mode in production. (Though I am kind of surprised how many people do that).

You can achieve that by setting INSTALL_PROVIDERS_FROM_SOURCES variable to true

This has been already addressed in #13439 in 2.0.1

potiuk commented 3 years ago

@colinglaes - was my diagnosis correct? Did you manage to fix the problem?

potiuk commented 3 years ago

I am looking at similar problem so I might come back with it

colinglaes commented 3 years ago

@potiuk Thanks for following up, I didn't use that flag when installing. Let me purge my installation and go through the installation steps again, retain the command and output history from the terminal here to give you better context.

colinglaes commented 3 years ago

@potiuk I just went through the process again and have successfully got the scheduler and webserver running. I believe that the source of my issue was that i had some apache_airflow_providers_* installed under /usr/local/lib/python3.6/dist-packages and others under /home/cglaes/.local/lib/python3.6/site-packages from earlier botched attempts that i hadn't removed in my last installation attempt.

I did have some thoughts though as to how the documentation / error messages could be improved though

https://airflow.apache.org/docs/apache-airflow/stable/installation.html

pip upgrade --pip==20.2.4 is an invalid command

Messages from airflow upgrade_check

"fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.". I think it might be clearer to explicitly state that the config key should be set as fernet_key = None instead of saying it "must be explicitly set empty". Not everyone knows what empty means in this case (i could see many people setting it like fernet_key =)
"1. The following invalid keys were found in your airflow.cfgcomes up and recommends removing the deprecated keys, additionally it states "Please generate a pod_template_file by runningairflow generate_pod_template`", there isn't an explicit statement that running this command is conditional on having a Kubernetes setup. Maybe these checks could be separated to improve clarity.

Let me know if it would be helpful for you guys if i paste the output from my terminal

potiuk commented 3 years ago

@potiuk I just went through the process again and have successfully got the scheduler and webserver running. I believe that the source of my issue was that i had some apache_airflow_providers_* installed under /usr/local/lib/python3.6/dist-packages and others under /home/cglaes/.local/lib/python3.6/site-packages from earlier botched attempts that i hadn't removed in my last installation attempt.

Cool.

I did have some thoughts though as to how the documentation / error messages could be improved though

https://airflow.apache.org/docs/apache-airflow/stable/installation.html

pip upgrade --pip==20.2.4 is an invalid command

Yep. This is already fixed in the upcoming 2.0.1 version: https://github.com/apache/airflow/blob/master/docs/apache-airflow/installation.rst

Messages from airflow upgrade_check

"fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.". I think it might be clearer to explicitly state that the config key should be set as fernet_key = None instead of saying it "must be explicitly set empty". Not everyone knows what empty means in this case (i could see many people setting it like fernet_key =)

"1. The following invalid keys were found in your airflow.cfgcomes up and recommends removing the deprecated keys, additionally it states "Please generate a pod_template_file by runningairflow generate_pod_template`", there isn't an explicit statement that running this command is conditional on having a Kubernetes setup. Maybe these checks could be separated to improve clarity.

Let me know if it would be helpful for you guys if i paste the output from my terminal

Would you be so kind to open Pull Request fixing those messages/statements ?

The Upgrade check is here: https://github.com/apache/airflow/tree/v1-10-stable/airflow/upgrade and it would be great if you could make a PR to v1-10-stable branch to fix those.

apache / airflow

Airflow 2.0 - ModuleNotFoundError: No module named 'airflow.providers.http' #13603