Closed erwanj closed 2 years ago
Hello @erwanj
I don't think so. It seems that this is an Airflow issue on DAG serialization. I found a similar issue on Airflow repository: https://github.com/apache/airflow/issues/20875
I will investigate more on this.
Please, take a look to DAG serialization, disabling it, may be a temporal solution for you.
Please, try to run:
airflow dags reserialize
Thank you very much for your feedback. I have tried with pdi_flow.py from sample_dags folder. I have removed job2 and trans2 as I don't have a carte server. I have the same behaviour.
Broken DAG: [/opt/airflow/dags/pdi_flow.py] Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 578, in serialize_operator
serialize_op['params'] = cls._serialize_params_dict(op.params)
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 451, in _serialize_params_dict
if f'{v.__module__}.{v.__class__.__name__}' == 'airflow.models.param.Param':
AttributeError: 'str' object has no attribute '__module__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 939, in to_dict
json_dict = {"__version": cls.SERIALIZER_VERSION, "dag": cls.serialize_dag(var)}
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 851, in serialize_dag
raise SerializationError(f'Failed to serialize DAG {dag.dag_id!r}: {e}')
airflow.exceptions.SerializationError: Failed to serialize DAG 'pdi_flow': 'str' object has no attribute '__module__'
If I remove 'date': '{{ ds }}'
from params, I don't get the error.
If you have the possibility to do it, can you confirm me that this sample dag pdi_flow.py
does work on airflow v2.2.4.
Thank you.
it seems that it works now.
I had to :
from airflow.models.param import Param
and use the following syntax for params :
params={
'date': Param('{{ ds }}')
}
Thank you for your help
It seems that params
should not be used, because it's already beeing used by BaseOperator.
I found a conversation about this.
This argument property, params
, will be renamed to task_params
.
Hi @erwanj , I applied a patch and also I released v1.0.9. Please, update the plugin. Thank you for reporting this.
Thank you very much for your help.
we have just upgraded from airflow v1.10.11 to airflow v2.2.4. We are getting the following error in airflow. This dag was working correctly before the migration.
The dag looks like :
If we remove
"date": '{{ ds }}'
, the error in airflow disapears.Is it possible that the issue is linked with airflow-pentaho-plugin?