geosolutions-it / oaw-georectify-process-airflow

Airflow based project to manage the geoprectify process
1 stars 0 forks source link

DAG fails when metadata information is not provided #26

Closed chpicone closed 3 years ago

chpicone commented 3 years ago

Customer reported this issue:

The airflow thing works fine but I have a few Dags they don’t work (even if I trigger them again)
- KIIVBlOE196
- KVBlOE1488
- KVBlOE775
- KVBlOEHun29
- KVBlWE82

This is the stacktrace from the Airflow Log section:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/plugins/custom_operator/georectify.py", line 22, in execute
    return self._geonode_payload()
  File "/opt/airflow/plugins/custom_operator/georectify.py", line 53, in _geonode_payload
    "data_quality_statement": metadata.get('format', None).replace("+", " "),
AttributeError: 'NoneType' object has no attribute 'replace'

Taking a look into the code we can see the cause of the issue: https://github.com/geosolutions-it/oaw-georectify-process-airflow/blob/269ba16a082a3746925d9b4f97c9f383999b78a4/plugins/custom_operator/georectify.py#L53

When we dont have a metadata information the get method assign by default the None object and then we cannot use the replace method.

chpicone commented 3 years ago

We have 2 options:

I asked the customer to decide how to proceed.

chpicone commented 3 years ago

Customer wants to make all attributes mandatories. So the DAG has to fail but reporting a more clear message, like:

AttributeError: 'format' metadata attribute is missing!