search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.3k forks source link

AttributeError: 'tuple' object has no attribute 'items' when entering UI Admin Variables Section #43375

Closed matteosdocsity closed 2 weeks ago

matteosdocsity commented 3 weeks ago

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After re-creating the containers of both Airflow 2.10.2, Postgres 15.4, and Python 3.12, without any changes from previous configurations, I encountered an error when trying to access the Variables UI Panel.

airflow-1  | [2024-10-25T07:07:01.991+0000] {app.py:1744} ERROR - Exception on /variable/list/ [GET]
airflow-1  | Traceback (most recent call last):
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
airflow-1  |     response = self.full_dispatch_request()
airflow-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
airflow-1  |     rv = self.handle_user_exception(e)
airflow-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
airflow-1  |     rv = self.dispatch_request()
airflow-1  |          ^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
airflow-1  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask_appbuilder/security/decorators.py", line 137, in wraps
airflow-1  |     return f(self, *args, **kwargs)
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask_appbuilder/views.py", line 550, in list
airflow-1  |     widgets = self._list()
airflow-1  |               ^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask_appbuilder/baseviews.py", line 1186, in _list
airflow-1  |     form = self.search_form.refresh()
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask_appbuilder/forms.py", line 327, in refresh
airflow-1  |     form = self(obj=obj)
airflow-1  |            ^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/form.py", line 209, in __call__
airflow-1  |     return type.__call__(cls, *args, **kwargs)
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/flask_wtf/form.py", line 73, in __init__
airflow-1  |     super().__init__(formdata=formdata, **kwargs)
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/form.py", line 281, in __init__
airflow-1  |     super().__init__(self._unbound_fields, meta=meta_obj, prefix=prefix)
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/form.py", line 49, in __init__
airflow-1  |     field = meta.bind_field(self, unbound_field, options)
airflow-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/meta.py", line 28, in bind_field
airflow-1  |     return unbound_field.bind(form=form, **options)
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/fields/core.py", line 387, in bind
airflow-1  |     return self.field_class(*self.args, **kw)
airflow-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-1  |   File "/usr/local/lib/python3.12/site-packages/wtforms/fields/core.py", line 133, in __init__
airflow-1  |     for k, v in flags.items():
airflow-1  |                 ^^^^^^^^^^^
airflow-1  | AttributeError: 'tuple' object has no attribute 'items'

What you think should happen instead?

Accessing correctly to any airflow panel

How to reproduce

Re-create containers for Airflow 2.10.2, Postgres 15.4, and Python 3.12. Attempt to access the Variables UI Panel. Observe the error.

Operating System

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.0.0 apache-airflow-providers-common-compat==1.2.1 apache-airflow-providers-common-io==1.4.2 apache-airflow-providers-common-sql==1.18.0 apache-airflow-providers-fab==1.4.1 apache-airflow-providers-facebook==3.6.0 apache-airflow-providers-ftp==3.11.1 apache-airflow-providers-google==10.23.0 apache-airflow-providers-http==4.13.1 apache-airflow-providers-imap==3.7.0

Deployment

Docker-Compose

Deployment details

Here is the docker-compose.yml

services:
  postgres:
    platform: linux/arm64
    image: postgres:15.4
    restart: unless-stopped
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    volumes:
      - "pgsql-data:/var/lib/postgresql/data"
    ports:
      - "5432:5432"
  airflow:
    platform: linux/arm64
    build:
      context: .
      dockerfile: ./src/airflow/Dockerfile
    restart: unless-stopped
    depends_on:
      - postgres
    environment:
      - EXECUTOR=Local
      - AIRFLOW__CORE__FERNET_KEY=***
      - AIRFLOW__WEBSERVER__WORKERS=4
      - AIRFLOW__WEBSERVER__BASE_URL=http://localhost:8080
      - GOOGLE_APPLICATION_CREDENTIALS=***
    volumes:
      - ./src/dags:/usr/local/airflow/dags
      - ./src/utils:/usr/local/airflow/utils
      - ./confs:/usr/local/airflow/confs
      - ./src/airflow:/usr/local/airflow
    env_file:
      - ./.env    
    ports:
      - "8080:8080"
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

volumes:
  pgsql-data:
    driver: local

Anything else?

Here the entire pip freeze

apache-airflow-providers-amazon==9.0.0
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.18.0
apache-airflow-providers-fab==1.4.1
apache-airflow-providers-facebook==3.6.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-google==10.23.0
apache-airflow-providers-http==4.13.1
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-mysql==5.7.1
apache-airflow-providers-postgres==5.13.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1
root@c9f1c90f96d2:/usr/local/airflow# pip freeze
aiofiles==23.2.1
aiohappyeyeballs==2.4.3
aiohttp==3.10.10
aiosignal==1.3.1
airflow-add-ons==0.2.14
alembic==1.13.3
annotated-types==0.7.0
anyio==4.6.2.post1
apache-airflow==2.10.2
apache-airflow-providers-amazon==9.0.0
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.18.0
apache-airflow-providers-fab==1.4.1
apache-airflow-providers-facebook==3.6.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-google==10.23.0
apache-airflow-providers-http==4.13.1
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-mysql==5.7.1
apache-airflow-providers-postgres==5.13.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1
apispec==6.7.0
argcomplete==3.5.1
asana==5.0.10
asgiref==3.8.1
asn1crypto==1.5.1
attrs==24.2.0
Authlib==1.3.2
babel==2.16.0
backoff==2.2.1
bcrypt==4.2.0
beautifulsoup4==4.12.3
blinker==1.8.2
boto3==1.35.48
botocore==1.35.48
Brotli==1.1.0
cachelib==0.9.0
cachetools==5.3.3
cattrs==24.1.2
certifi==2024.8.30
cffi==1.17.1
chardet==5.2.0
charset-normalizer==3.4.0
cli_exit_tools==1.2.7
click==8.1.7
clickclick==20.10.2
colorama==0.4.6
colorlog==6.8.2
ConfigUpdater==3.2
connexion==2.14.2
cramjam==2.9.0
cron-descriptor==1.4.5
croniter==3.0.3
cryptography==43.0.3
curlify==2.2.1
customerio==2.1
Cython==3.0.11
db-dtypes==1.3.0
decorator==5.1.1
Deprecated==1.2.14
dill==0.3.9
dnspython==2.7.0
docstring_parser==0.16
docutils==0.21.2
email_validator==2.2.0
emoji==2.13.2
et-xmlfile==1.1.0
facebook-sdk==3.1.0
facebook_business==20.0.3
facebookads==2.11.4
fastparquet==2024.5.0
fasttext @ git+https://github.com/pualien/fastText@4a3165b25e8000a880818b44c54bd1b2b5504fd2
fasttext-langdetect==1.0.5
filelock==3.16.1
Flask==2.2.5
Flask-AppBuilder==4.5.0
Flask-Babel==2.0.0
Flask-Caching==2.3.0
Flask-JWT-Extended==4.6.0
Flask-Limiter==3.8.0
Flask-Login==0.6.3
Flask-Session==0.5.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.2.2
frozenlist==1.5.0
fsspec==2024.10.0
gcloud-aio-auth==5.3.2
gcloud-aio-bigquery==7.1.0
gcloud-aio-storage==9.3.0
gcloud-connectors==0.2.2
gcsfs==2024.10.0
google-ads==25.1.0
google-analytics-admin==0.23.1
google-analytics-data==0.18.12
google-api-core==2.21.0
google-api-python-client==2.90.0
google-auth==2.35.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.1
google-cloud-aiplatform==1.70.0
google-cloud-appengine-logging==1.5.0
google-cloud-audit-log==0.3.0
google-cloud-automl==2.14.0
google-cloud-batch==0.17.30
google-cloud-bigquery==3.26.0
google-cloud-bigquery-datatransfer==3.17.0
google-cloud-bigquery-storage==2.27.0
google-cloud-bigtable==2.26.0
google-cloud-build==3.26.0
google-cloud-compute==1.20.0
google-cloud-container==2.53.0
google-cloud-core==2.4.1
google-cloud-datacatalog==3.21.0
google-cloud-dataflow-client==0.8.13
google-cloud-dataform==0.5.12
google-cloud-dataplex==2.3.0
google-cloud-dataproc==5.15.0
google-cloud-dataproc-metastore==1.16.0
google-cloud-dlp==3.25.0
google-cloud-kms==3.1.0
google-cloud-language==2.15.0
google-cloud-logging==3.11.3
google-cloud-memcache==1.10.0
google-cloud-monitoring==2.23.0
google-cloud-orchestration-airflow==1.15.0
google-cloud-os-login==2.15.0
google-cloud-pubsub==2.25.2
google-cloud-redis==2.16.0
google-cloud-resource-manager==1.13.0
google-cloud-run==0.10.10
google-cloud-secret-manager==2.21.0
google-cloud-spanner==3.49.1
google-cloud-speech==2.28.0
google-cloud-storage==2.18.2
google-cloud-storage-transfer==1.13.0
google-cloud-tasks==2.17.0
google-cloud-texttospeech==2.20.0
google-cloud-translate==3.17.0
google-cloud-videointelligence==2.14.0
google-cloud-vision==3.8.0
google-cloud-workflows==1.15.0
google-crc32c==1.6.0
google-re2==1.1.20240702
google-resumable-media==2.7.2
googleads==43.0.0
googleapis-common-protos==1.65.0
greenlet==3.1.1
grpc-google-iam-v1==0.13.1
grpc-interceptor==0.15.4
grpcio==1.67.0
grpcio-gcp==0.2.2
grpcio-status==1.62.3
grpcio-tools==1.62.3
gspread==5.4.0
gspread-pandas==3.2.2
gunicorn==23.0.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==1.0.6
httplib2==0.22.0
httpx==0.27.2
hyperframe==6.0.1
idna==3.10
IMAPClient==2.1.0
immutabledict==4.2.0
importlib_metadata==8.4.0
importlib_resources==6.4.5
inflection==0.5.1
iniconfig==2.0.0
isodate==0.7.2
itsdangerous==2.2.0
jaraco.classes==3.4.0
jaraco.context==6.0.1
jaraco.functools==4.1.0
jeepney==0.8.0
Jinja2==3.1.4
jmespath==1.0.1
json-merge-patch==0.2
jsonpath-ng==1.7.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jwplatform==2.2.2
keyring==25.3.0
lazy-object-proxy==1.10.0
lib-detect-testenv==2.0.8
limits==3.13.0
linkify-it-py==2.0.3
lockfile==0.12.2
looker-sdk==24.16.2
lxml==5.3.0
Mako==1.3.6
markdown-it-py==3.0.0
MarkupSafe==3.0.2
marshmallow==3.23.0
marshmallow-oneofschema==3.1.1
marshmallow-sqlalchemy==0.28.2
mdit-py-plugins==0.4.2
mdurl==0.1.2
methodtools==0.4.7
mock==5.1.0
more-itertools==10.5.0
multidict==6.1.0
multiprocess==0.70.17
mysql-connector-python==9.1.0
mysqlclient==2.2.5
neterr==1.1.1
numpy==1.26.4
oauth2client==4.1.3
oauthlib==3.2.2
openpyxl==3.1.5
opentelemetry-api==1.27.0
opentelemetry-exporter-otlp==1.27.0
opentelemetry-exporter-otlp-proto-common==1.27.0
opentelemetry-exporter-otlp-proto-grpc==1.27.0
opentelemetry-exporter-otlp-proto-http==1.27.0
opentelemetry-proto==1.27.0
opentelemetry-sdk==1.27.0
opentelemetry-semantic-conventions==0.48b0
ordered-set==4.1.0
ordereddict==1.1
orjson==3.10.7
packaging==24.1
pandas==2.1.4
pandas-gbq==0.24.0
paramiko==3.5.0
pathspec==0.12.1
pendulum==3.0.0
platformdirs==4.3.6
pluggy==1.5.0
ply==3.11
portalocker==2.10.1
prison==0.2.1
propcache==0.2.0
proto-plus==1.25.0
protobuf==4.25.5
psutil==6.1.0
psycopg2-binary==2.9.10
py==1.11.0
pyarrow==17.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.0
PyAthena==3.9.0
pybind11==2.13.6
pycountry==24.6.1
pycparser==2.22
pydantic==2.9.2
pydantic_core==2.23.4
pydata-google-auth==1.8.2
Pygments==2.18.0
PyJWT==2.9.0
pymongo==4.10.1
pymongo-ssh==0.0.15
PyMySQL==1.1.0
PyNaCl==1.5.0
pyOpenSSL==24.2.1
pyparsing==3.2.0
pytest==8.3.3
python-daemon==3.1.0
python-dateutil==2.9.0.post0
python-nvd3==0.16.0
python-slugify==8.0.4
python3-saml==1.16.0
pytz==2024.2
PyYAML==6.0.1
qdrant-client==1.12.0
redshift-connector==2.1.3
referencing==0.35.1
requests==2.32.3
requests-file==2.1.0
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
retry==0.9.2
rfc3339-validator==0.1.4
rich==13.9.3
rich-argparse==1.5.2
rpds-py==0.20.0
rsa==4.9
s3transfer==0.10.3
scramp==1.4.5
SecretStorage==3.3.3
sentry-sdk==2.17.0
setproctitle==1.3.3
setuptools==75.2.0
shapely==2.0.6
simplejson==3.19.3
six==1.16.0
slack_sdk==3.33.2
sniffio==1.3.1
soupsieve==2.6
SQLAlchemy==1.4.54
sqlalchemy-bigquery==1.12.0
sqlalchemy-connector==0.1.43
SQLAlchemy-JSONField==1.0.2
sqlalchemy-redshift==0.8.14
sqlalchemy-spanner==1.7.0
SQLAlchemy-Utils==0.41.2
sqlparse==0.5.1
sshtunnel==0.4.0
swagger-ui-bundle==0.0.9
tabulate==0.9.0
tenacity==9.0.0
termcolor==2.5.0
text-unidecode==1.3
time-machine==2.16.0
timeout-decorator==0.5.0
tldextract==5.1.2
toml==0.10.2
tornado==6.4.1
typing_extensions==4.12.2
tzdata==2024.2
uc-micro-py==1.0.3
unbounce-python-api==1.3.5
unicodecsv==0.14.1
universal_pathlib==0.2.5
uritemplate==4.1.1
urllib3==2.2.3
usercustomize==1.0.0
watchtower==3.3.1
Werkzeug==2.2.3
wheel==0.44.0
wirerope==0.4.7
wrapt==1.16.0
wrapt-timeout-decorator==1.3.12.2
WTForms==3.2.1
xlrd==1.2.0
xmlsec==1.3.14
xmltodict==0.14.2
yarl==1.16.0
zeep==4.3.1
zipp==3.20.2

Are you willing to submit PR?

Code of Conduct

potiuk commented 2 weeks ago

Duplicate of #43356 . You have not used constraints https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html when installing Airflow (please do as we recommend), this is not an airflow issue - this is FAB issue that is already fixed and 2.10.3 will be using the fixed version.

matteosdocsity commented 2 weeks ago

I understand this is a consequence of the version of apache-airflow-providers-fab , but indeed it's related on wrong and NOT tested constraint for WTForms==3.2.0|3.2.1..btw taking attention on logs it was quite easy to fix, but maybe something in the process of releases and testing was not a complete E2E testing, I'm guessing. I don't want to be polemical, but to highlight the issue

potiuk commented 2 weeks ago

No. You are entirely wrong @matteosdocsity . the constraints you linked to show WTForms 3.1.2 and the broken ones are 3.2.0 and 3.2.1 . You mixed numbers.

So it's you who made a mistake by looking up the numbers wrong, not testing.

Next time be a little more diligent please - being not polemical makes sense when you do double checking.

matteosdocsity commented 2 weeks ago

Oh ok sorry @potiuk you're right I have not tested forcing the installation of WTForms 3.1.2...the mistake was the release not tested of the fab provider, got it!