search

sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

AUB Fails (OAI) #164

Closed aaron-collier closed 2 years ago

aaron-collier commented 2 years ago 
[2022-08-16, 20:37:06 UTC] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dlme_airflow/harvester/source_harvester.py", line 17, in data_source_harvester
    dataframe_to_file(collection)
  File "/opt/airflow/dlme_airflow/utils/dataframe.py", line 26, in dataframe_to_file
    working_csv = os.path.join(root_dir, "working", data_path, "data.csv")
  File "/usr/local/lib/python3.9/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/local/lib/python3.9/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'
aaron-collier commented 2 years ago

In my current testing, AUB succeeds all the way though indexing. This may be closable, but will run a few more final tests.