search

sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

Princeton papyri collectin failing index #387

Closed jacobthill closed 1 year ago

jacobthill commented 1 year ago 

  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 175, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 193, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dlme_airflow/tasks/index.py", line 25, in index_collection
    return response.json()["message"]
KeyError: 'message'```

This is probably a data error somewhere earlier in the DAG, maybe there are invalid urls in the IR or duplicate ids, etc.