astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
348 stars 43 forks source link

Example dag failing with raise ImportError(f"{classname} was not found in allow list for import") #1677

Open kelechi2020 opened 1 year ago

kelechi2020 commented 1 year ago

Describe the bug Example dag which used to pass now fails failing with error on airflow v2.5.1

Version

To Reproduce Steps to reproduce the behavior:

  1. Write the DAG calculate_popular_movies.py
  2. Create connection '.postgres_conn'
  3. See error
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/abstractoperator.py", line 602, in _do_render_template_fields
    rendered_content = self.render_template(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/abstractoperator.py", line 676, in render_template
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/abstractoperator.py", line 676, in <dictcomp>
    return {k: self.render_template(v, context, jinja_env, oids) for k, v in value.items()}
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/abstractoperator.py", line 666, in render_template
    return value.resolve(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/xcom_arg.py", line 331, in resolve
    result = ti.xcom_pull(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 2367, in xcom_pull
    return XCom.deserialize_value(first)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/xcom.py", line 661, in deserialize_value
    return BaseXCom._deserialize_value(result, False)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/xcom.py", line 654, in _deserialize_value
    return json.loads(result.value.decode("UTF-8"), cls=XComDecoder, object_hook=object_hook)
  File "/usr/local/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/json.py", line 215, in object_hook
    raise ImportError(f"{classname} was not found in allow list for import")

Screenshots

image

Additional context Add any other context about the problem here.

tswinb commented 1 year ago

I was able to fix this by adding AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES: airflow\.* astro\.* to my .env file. Sourced from this tutorial: https://docs.astronomer.io/learn/astro-python-sdk (but this instruction seems to be missing from the ETL tutorial https://docs.astronomer.io/learn/astro-python-sdk-etl ).