search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
248 stars 75 forks source link

airflow v2.1.3 webserver failing to start after installing latest airflow-exporter #107

Closed pindge closed 2 years ago

pindge commented 3 years ago

After installing airflow-exporter airflow version 2.1.3 shifted to lower than 2.1.0 version

+ kubectl logs airflow-web-85bc665c56-5jjjr -n processing -c install-pip-packages
Requirement already satisfied: boto3 in /home/airflow/.local/lib/python3.8/site-packages (1.17.112)
Requirement already satisfied: kubernetes in /home/airflow/.local/lib/python3.8/site-packages (11.0.0)
Collecting airflow-kubernetes-job-operator
  Downloading airflow_kubernetes_job_operator-2.0.3-py2.py3-none-any.whl (47 kB)
Collecting SQLAlchemy==1.3.23
  Downloading SQLAlchemy-1.3.23-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting airflow-exporter
  Downloading airflow_exporter-1.5.2-py2.py3-none-any.whl (6.3 kB)
Collecting authlib
  Downloading Authlib-0.15.5-py2.py3-none-any.whl (203 kB)
Collecting Flask-AppBuilder==3.1.1
  Downloading Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
Requirement already satisfied: python-dateutil<3,>=2.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (2.8.2)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.4 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (2.5.1)
Requirement already satisfied: Flask-Login<0.5,>=0.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.4.1)
Requirement already satisfied: marshmallow-enum<2,>=1.5.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.5.1)
Requirement already satisfied: marshmallow-sqlalchemy<0.24.0,>=0.22.0 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.23.1)
Requirement already satisfied: PyJWT>=1.7.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.7.1)
Requirement already satisfied: sqlalchemy-utils<1,>=0.32.21 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.37.8)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.25.1)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.3.0)
Requirement already satisfied: Flask-Babel<2,>=1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.0.0)
Requirement already satisfied: prison<1.0.0,>=0.1.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.1.3)
Requirement already satisfied: colorama<1,>=0.3.9 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.4.4)
Requirement already satisfied: apispec[yaml]<4,>=3.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.3.2)
Requirement already satisfied: marshmallow<4,>=3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.13.0)
Requirement already satisfied: Flask<2,>=0.12 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.1.4)
Requirement already satisfied: email-validator<2,>=1.0.5 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.1.3)
Requirement already satisfied: click<8,>=6.7 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (7.1.2)
Requirement already satisfied: jsonschema<4,>=3.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.2.0)
Requirement already satisfied: Flask-WTF<0.15.0,>=0.14.2 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.14.3)
Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (0.4.2)
Requirement already satisfied: botocore<1.21.0,>=1.20.112 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (1.20.112)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (0.10.0)
Requirement already satisfied: urllib3>=1.24.2 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.26.6)
Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.8/site-packages (from kubernetes) (57.5.0)
Requirement already satisfied: pyyaml>=3.12 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (5.4.1)
Requirement already satisfied: six>=1.9.0 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.16.0)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.2.1)
Requirement already satisfied: requests-oauthlib in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.3.0)
Requirement already satisfied: google-auth>=1.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.35.0)
Requirement already satisfied: certifi>=14.05.14 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (2020.12.5)
Requirement already satisfied: requests in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (2.26.0)
Collecting zthreading>=0.1.13
  Downloading zthreading-0.1.17-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: apache-airflow>=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from airflow-exporter) (2.1.3)
Requirement already satisfied: prometheus-client>=0.4.2 in /home/airflow/.local/lib/python3.8/site-packages (from airflow-exporter) (0.8.0)
Requirement already satisfied: cryptography in /home/airflow/.local/lib/python3.8/site-packages (from authlib) (3.4.7)
Requirement already satisfied: markdown<4.0,>=2.5.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (3.3.4)
Requirement already satisfied: apache-airflow-providers-sqlite in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.0.0)
Requirement already satisfied: colorlog>=4.0.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (4.8.0)
Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (5.8.0)
Requirement already satisfied: pandas<2.0,>=0.17.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.3.2)
Requirement already satisfied: argcomplete~=1.10 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.12.3)
Requirement already satisfied: pendulum~=2.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.1.2)
Requirement already satisfied: tabulate<0.9,>=0.7.5 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.8.9)
Requirement already satisfied: swagger-ui-bundle>=0.0.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.0.8)
Requirement already satisfied: unicodecsv>=0.14.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.14.1)
Requirement already satisfied: docutils<0.17 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.16)
Requirement already satisfied: iso8601>=0.1.12 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.1.16)
Requirement already satisfied: markupsafe<2.0,>=1.1.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.1.1)
Requirement already satisfied: flask-caching<2.0.0,>=1.5.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.10.1)
Requirement already satisfied: inflection>=0.3.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.5.1)
Requirement already satisfied: clickclick>=1.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (20.10.2)
Requirement already satisfied: importlib-resources~=1.4 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.5.0)
Requirement already satisfied: croniter<1.1,>=0.3.17 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.0.15)
Requirement already satisfied: termcolor>=1.1.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.1.0)
Requirement already satisfied: marshmallow-oneofschema>=2.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (3.0.1)
Requirement already satisfied: sqlalchemy-jsonfield~=1.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.0.0)
Requirement already satisfied: apache-airflow-providers-ftp in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.0.0)
Requirement already satisfied: python-nvd3~=0.15.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.15.0)
Requirement already satisfied: pygments<3.0,>=2.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.10.0)
Requirement already satisfied: importlib-metadata>=1.7 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (4.6.4)
Requirement already satisfied: lazy-object-proxy in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.4.3)
Requirement already satisfied: apache-airflow-providers-imap in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.0.0)
Requirement already satisfied: werkzeug>=1.0.1,~=1.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.0.1)
Requirement already satisfied: rich>=9.2.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (10.7.0)
Requirement already satisfied: alembic<2.0,>=1.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.6.5)
Requirement already satisfied: lockfile>=0.12.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.12.2)
Requirement already satisfied: setproctitle<2,>=1.1.8 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.2.2)
Requirement already satisfied: python3-openid~=3.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (3.2.0)
Collecting apache-airflow>=2.0.0
  Downloading apache_airflow-2.2.2-py3-none-any.whl (5.3 MB)
Requirement already satisfied: tenacity>=6.2.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (6.2.0)
Requirement already satisfied: packaging>=14.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (20.9)
Requirement already satisfied: wtforms<3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.3.3)
Requirement already satisfied: gunicorn>=20.1.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (20.1.0)
  Downloading apache_airflow-2.2.1-py3-none-any.whl (5.3 MB)
  Downloading apache_airflow-2.2.0-py3-none-any.whl (5.3 MB)
  Downloading apache_airflow-2.1.4-py3-none-any.whl (5.3 MB)
  Downloading apache_airflow-2.1.2-py3-none-any.whl (5.2 MB)
  Downloading apache_airflow-2.1.1-py3-none-any.whl (5.2 MB)
  Downloading apache_airflow-2.1.0-py3-none-any.whl (5.3 MB)
Requirement already satisfied: cattrs~=1.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.5.0)
  Downloading apache_airflow-2.0.2-py3-none-any.whl (4.6 MB)
Collecting connexion[flask,swagger-ui]<3,>=2.6.0
  Downloading connexion-2.9.0-py2.py3-none-any.whl (84 kB)
Collecting gunicorn<20.0,>=19.5.0
  Downloading gunicorn-19.10.0-py2.py3-none-any.whl (113 kB)
Collecting croniter<0.4,>=0.3.17
  Downloading croniter-0.3.37-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: itsdangerous>=1.1.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.1.0)
Collecting rich==9.2.0
  Downloading rich-9.2.0-py3-none-any.whl (164 kB)
Requirement already satisfied: python-slugify<5.0,>=3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (4.0.1)
Requirement already satisfied: python-daemon>=2.2.4 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.3.0)
Collecting importlib-metadata~=1.7
  Downloading importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
Requirement already satisfied: graphviz>=0.12 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.17)
Requirement already satisfied: numpy in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.20.3)
Requirement already satisfied: jinja2<2.12.0,>=2.10.1 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.11.3)
Requirement already satisfied: blinker in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (1.4)
Requirement already satisfied: attrs<21.0,>=20.0 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (20.3.0)
Requirement already satisfied: apache-airflow-providers-http in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (2.0.0)
Collecting cached-property~=1.5
  Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Requirement already satisfied: dill<0.4,>=0.2.2 in /home/airflow/.local/lib/python3.8/site-packages (from apache-airflow>=2.0.0->airflow-exporter) (0.3.1.1)
Requirement already satisfied: typing-extensions<4.0.0,>=3.7.4 in /home/airflow/.local/lib/python3.8/site-packages (from rich==9.2.0->apache-airflow>=2.0.0->airflow-exporter) (3.7.4.3)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /home/airflow/.local/lib/python3.8/site-packages (from rich==9.2.0->apache-airflow>=2.0.0->airflow-exporter) (0.9.1)
Requirement already satisfied: python-editor>=0.3 in /home/airflow/.local/lib/python3.8/site-packages (from alembic<2.0,>=1.2->apache-airflow>=2.0.0->airflow-exporter) (1.0.4)
Requirement already satisfied: Mako in /home/airflow/.local/lib/python3.8/site-packages (from alembic<2.0,>=1.2->apache-airflow>=2.0.0->airflow-exporter) (1.1.4)
Requirement already satisfied: openapi-spec-validator<0.4,>=0.2.4 in /home/airflow/.local/lib/python3.8/site-packages (from connexion[flask,swagger-ui]<3,>=2.6.0->apache-airflow>=2.0.0->airflow-exporter) (0.3.1)
Collecting natsort
  Downloading natsort-8.0.0-py3-none-any.whl (37 kB)
Requirement already satisfied: cffi>=1.12 in /home/airflow/.local/lib/python3.8/site-packages (from cryptography->authlib) (1.14.6)
Requirement already satisfied: pycparser in /home/airflow/.local/lib/python3.8/site-packages (from cffi>=1.12->cryptography->authlib) (2.20)
Requirement already satisfied: dnspython>=1.15.0 in /home/airflow/.local/lib/python3.8/site-packages (from email-validator<2,>=1.0.5->Flask-AppBuilder==3.1.1) (1.16.0)
Requirement already satisfied: idna>=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from email-validator<2,>=1.0.5->Flask-AppBuilder==3.1.1) (3.2)
Requirement already satisfied: pytz in /home/airflow/.local/lib/python3.8/site-packages (from Flask-Babel<2,>=1->Flask-AppBuilder==3.1.1) (2021.1)
Requirement already satisfied: Babel>=2.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-Babel<2,>=1->Flask-AppBuilder==3.1.1) (2.9.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (4.2.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (0.2.8)
Requirement already satisfied: zipp>=0.5 in /home/airflow/.local/lib/python3.8/site-packages (from importlib-metadata~=1.7->apache-airflow>=2.0.0->airflow-exporter) (3.5.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/airflow/.local/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->Flask-AppBuilder==3.1.1) (0.18.0)
Requirement already satisfied: openapi-schema-validator in /home/airflow/.local/lib/python3.8/site-packages (from openapi-spec-validator<0.4,>=0.2.4->connexion[flask,swagger-ui]<3,>=2.6.0->apache-airflow>=2.0.0->airflow-exporter) (0.1.5)
Requirement already satisfied: pytzdata>=2020.1 in /home/airflow/.local/lib/python3.8/site-packages (from pendulum~=2.0->apache-airflow>=2.0.0->airflow-exporter) (2020.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/airflow/.local/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes) (0.4.8)
Requirement already satisfied: text-unidecode>=1.3 in /home/airflow/.local/lib/python3.8/site-packages (from python-slugify<5.0,>=3.0.0->apache-airflow>=2.0.0->airflow-exporter) (1.3)
Requirement already satisfied: defusedxml in /home/airflow/.local/lib/python3.8/site-packages (from python3-openid~=3.2->apache-airflow>=2.0.0->airflow-exporter) (0.7.1)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from requests->kubernetes) (2.0.4)
INFO: pip is looking at multiple versions of apache-airflow-providers-http to determine which version is compatible with other requirements. This could take a while.
Collecting apache-airflow-providers-http
  Downloading apache_airflow_providers_http-2.0.1-py3-none-any.whl (21 kB)
Requirement already satisfied: isodate in /home/airflow/.local/lib/python3.8/site-packages (from openapi-schema-validator->openapi-spec-validator<0.4,>=0.2.4->connexion[flask,swagger-ui]<3,>=2.6.0->apache-airflow>=2.0.0->airflow-exporter) (0.6.0)
Requirement already satisfied: oauthlib>=3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from requests-oauthlib->kubernetes) (3.1.1)
Installing collected packages: SQLAlchemy, natsort, connexion, rich, importlib-metadata, gunicorn, Flask-AppBuilder, croniter, cached-property, apache-airflow-providers-http, zthreading, apache-airflow, authlib, airflow-kubernetes-job-operator, airflow-exporter
  Attempting uninstall: SQLAlchemy
    Found existing installation: SQLAlchemy 1.3.24
    Uninstalling SQLAlchemy-1.3.24:
      Successfully uninstalled SQLAlchemy-1.3.24
  Attempting uninstall: rich
    Found existing installation: rich 10.7.0
    Uninstalling rich-10.7.0:
      Successfully uninstalled rich-10.7.0
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 4.6.4
    Uninstalling importlib-metadata-4.6.4:
      Successfully uninstalled importlib-metadata-4.6.4
  Attempting uninstall: gunicorn
    Found existing installation: gunicorn 20.1.0
    Uninstalling gunicorn-20.1.0:
      Successfully uninstalled gunicorn-20.1.0
  Attempting uninstall: Flask-AppBuilder
    Found existing installation: Flask-AppBuilder 3.3.2
    Uninstalling Flask-AppBuilder-3.3.2:
      Successfully uninstalled Flask-AppBuilder-3.3.2
  Attempting uninstall: croniter
    Found existing installation: croniter 1.0.15
    Uninstalling croniter-1.0.15:
      Successfully uninstalled croniter-1.0.15
  Attempting uninstall: apache-airflow-providers-http
    Found existing installation: apache-airflow-providers-http 2.0.0
    Uninstalling apache-airflow-providers-http-2.0.0:
      Successfully uninstalled apache-airflow-providers-http-2.0.0
  Attempting uninstall: apache-airflow
    Found existing installation: apache-airflow 2.1.3
    Uninstalling apache-airflow-2.1.3:
      Successfully uninstalled apache-airflow-2.1.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow-providers-ssh 2.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-slack 4.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-sftp 2.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-redis 2.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-postgres 2.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-mysql 2.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-microsoft-azure 3.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-hashicorp 2.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-grpc 2.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-google 5.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-elasticsearch 2.0.2 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-docker 2.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-cncf-kubernetes 2.0.2 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-celery 2.0.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
apache-airflow-providers-amazon 2.1.0 requires apache-airflow>=2.1.0, but you have apache-airflow 2.0.2 which is incompatible.
Successfully installed Flask-AppBuilder-3.1.1 SQLAlchemy-1.3.23 airflow-exporter-1.5.2 airflow-kubernetes-job-operator-2.0.3 apache-airflow-2.0.2 apache-airflow-providers-http-2.0.1 authlib-0.15.5 cached-property-1.5.2 connexion-2.9.0 croniter-0.3.37 gunicorn-19.10.0 importlib-metadata-1.7.0 natsort-8.0.0 rich-9.2.0 zthreading-0.1.17
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
copying '/home/airflow/.local/*' to '/opt/home-airflow-local'...

For comparison below is log with no airflow-exporter

(base) ubuntu@dea-dev-eks:~/datakube-apps$ k logs airflow-web-58d9cc69f8-mgjt4 -n processing -c install-pip-packages
+ kubectl logs airflow-web-58d9cc69f8-mgjt4 -n processing -c install-pip-packages
Requirement already satisfied: boto3 in /home/airflow/.local/lib/python3.8/site-packages (1.17.112)
Requirement already satisfied: kubernetes in /home/airflow/.local/lib/python3.8/site-packages (11.0.0)
Collecting airflow-kubernetes-job-operator
  Downloading airflow_kubernetes_job_operator-2.0.3-py2.py3-none-any.whl (47 kB)
Collecting SQLAlchemy==1.3.23
  Downloading SQLAlchemy-1.3.23-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting authlib
  Downloading Authlib-0.15.5-py2.py3-none-any.whl (203 kB)
Collecting Flask-AppBuilder==3.1.1
  Downloading Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
Requirement already satisfied: click<8,>=6.7 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (7.1.2)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.25.1)
Requirement already satisfied: python-dateutil<3,>=2.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (2.8.2)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.4 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (2.5.1)
Requirement already satisfied: prison<1.0.0,>=0.1.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.1.3)
Requirement already satisfied: jsonschema<4,>=3.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.2.0)
Requirement already satisfied: marshmallow-sqlalchemy<0.24.0,>=0.22.0 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.23.1)
Requirement already satisfied: Flask-Babel<2,>=1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.0.0)
Requirement already satisfied: PyJWT>=1.7.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.7.1)
Requirement already satisfied: Flask<2,>=0.12 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.1.4)
Requirement already satisfied: email-validator<2,>=1.0.5 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.1.3)
Requirement already satisfied: apispec[yaml]<4,>=3.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.3.2)
Requirement already satisfied: Flask-Login<0.5,>=0.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.4.1)
Requirement already satisfied: colorama<1,>=0.3.9 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.4.4)
Requirement already satisfied: sqlalchemy-utils<1,>=0.32.21 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.37.8)
Requirement already satisfied: Flask-WTF<0.15.0,>=0.14.2 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (0.14.3)
Requirement already satisfied: marshmallow-enum<2,>=1.5.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.5.1)
Requirement already satisfied: marshmallow<4,>=3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (3.13.0)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-AppBuilder==3.1.1) (1.3.0)
Requirement already satisfied: botocore<1.21.0,>=1.20.112 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (1.20.112)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (0.10.0)
Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /home/airflow/.local/lib/python3.8/site-packages (from boto3) (0.4.2)
Requirement already satisfied: urllib3>=1.24.2 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.26.6)
Requirement already satisfied: google-auth>=1.0.1 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.35.0)
Requirement already satisfied: requests in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (2.26.0)
Requirement already satisfied: six>=1.9.0 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.16.0)
Requirement already satisfied: requests-oauthlib in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.3.0)
Requirement already satisfied: certifi>=14.05.14 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (2020.12.5)
Requirement already satisfied: pyyaml>=3.12 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (5.4.1)
Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.8/site-packages (from kubernetes) (57.5.0)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /home/airflow/.local/lib/python3.8/site-packages (from kubernetes) (1.2.1)
Collecting zthreading>=0.1.13
  Downloading zthreading-0.1.17-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: cryptography in /home/airflow/.local/lib/python3.8/site-packages (from authlib) (3.4.7)
Requirement already satisfied: dnspython>=1.15.0 in /home/airflow/.local/lib/python3.8/site-packages (from email-validator<2,>=1.0.5->Flask-AppBuilder==3.1.1) (1.16.0)
Requirement already satisfied: idna>=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from email-validator<2,>=1.0.5->Flask-AppBuilder==3.1.1) (3.2)
Requirement already satisfied: itsdangerous<2.0,>=0.24 in /home/airflow/.local/lib/python3.8/site-packages (from Flask<2,>=0.12->Flask-AppBuilder==3.1.1) (1.1.0)
Requirement already satisfied: Jinja2<3.0,>=2.10.1 in /home/airflow/.local/lib/python3.8/site-packages (from Flask<2,>=0.12->Flask-AppBuilder==3.1.1) (2.11.3)
Requirement already satisfied: Werkzeug<2.0,>=0.15 in /home/airflow/.local/lib/python3.8/site-packages (from Flask<2,>=0.12->Flask-AppBuilder==3.1.1) (1.0.1)
Requirement already satisfied: Babel>=2.3 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-Babel<2,>=1->Flask-AppBuilder==3.1.1) (2.9.1)
Requirement already satisfied: pytz in /home/airflow/.local/lib/python3.8/site-packages (from Flask-Babel<2,>=1->Flask-AppBuilder==3.1.1) (2021.1)
Requirement already satisfied: python3-openid>=2.0 in /home/airflow/.local/lib/python3.8/site-packages (from Flask-OpenID<2,>=1.2.5->Flask-AppBuilder==3.1.1) (3.2.0)
Requirement already satisfied: WTForms in /home/airflow/.local/lib/python3.8/site-packages (from Flask-WTF<0.15.0,>=0.14.2->Flask-AppBuilder==3.1.1) (2.3.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/airflow/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes) (4.7.2)
Requirement already satisfied: MarkupSafe>=0.23 in /home/airflow/.local/lib/python3.8/site-packages (from Jinja2<3.0,>=2.10.1->Flask<2,>=0.12->Flask-AppBuilder==3.1.1) (1.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/airflow/.local/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->Flask-AppBuilder==3.1.1) (0.18.0)
Requirement already satisfied: attrs>=17.4.0 in /home/airflow/.local/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->Flask-AppBuilder==3.1.1) (20.3.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/airflow/.local/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes) (0.4.8)
Requirement already satisfied: defusedxml in /home/airflow/.local/lib/python3.8/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->Flask-AppBuilder==3.1.1) (0.7.1)
Requirement already satisfied: cffi>=1.12 in /home/airflow/.local/lib/python3.8/site-packages (from cryptography->authlib) (1.14.6)
Requirement already satisfied: pycparser in /home/airflow/.local/lib/python3.8/site-packages (from cffi>=1.12->cryptography->authlib) (2.20)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from requests->kubernetes) (2.0.4)
Requirement already satisfied: oauthlib>=3.0.0 in /home/airflow/.local/lib/python3.8/site-packages (from requests-oauthlib->kubernetes) (3.1.1)
Installing collected packages: SQLAlchemy, zthreading, Flask-AppBuilder, authlib, airflow-kubernetes-job-operator
  Attempting uninstall: SQLAlchemy
    Found existing installation: SQLAlchemy 1.3.24
    Uninstalling SQLAlchemy-1.3.24:
      Successfully uninstalled SQLAlchemy-1.3.24
  Attempting uninstall: Flask-AppBuilder
    Found existing installation: Flask-AppBuilder 3.3.2
    Uninstalling Flask-AppBuilder-3.3.2:
      Successfully uninstalled Flask-AppBuilder-3.3.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow 2.1.3 requires flask-appbuilder<4.0.0,>=3.3.2, but you have flask-appbuilder 3.1.1 which is incompatible.
Successfully installed Flask-AppBuilder-3.1.1 SQLAlchemy-1.3.23 airflow-kubernetes-job-operator-2.0.3 authlib-0.15.5 zthreading-0.1.17
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
copying '/home/airflow/.local/*' to '/opt/home-airflow-local'...
elephantum commented 3 years ago

@pindge what is the fastest way to reproduce this?

pindge commented 2 years ago

Closing, the real problem was flask-appbuilder 3.1.1

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow 2.1.3 requires flask-appbuilder<4.0.0,>=3.3.2, but you have flask-appbuilder 3.1.1 which is incompatible.