sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows
Apache License 2.0
5 stars 0 forks source link

Fix broken gobi dag, related to gather_files_task #985

Closed jgreben closed 2 months ago

jgreben commented 2 months ago

The gobi dag is erroring:

Broken DAG: [/opt/airflow/libsys_airflow/dags/data_exports/gobi_transmission.py]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/ast.py", line 74, in _convert_num
    _raise_malformed_node(node)
  File "/usr/local/lib/python3.11/ast.py", line 71, in _raise_malformed_node
    raise ValueError(msg + f': {node!r}')
ValueError: malformed node or string: XComArg(<Task(_PythonDecoratedOperator): gather_files_task>)

This seems to be related to the return value of gobi_list_from_marc_files in libsys_airflow/plugins/data_exports/marc/gobi.py because the transmit_data task in the gobi transmission dag (libsys_airflow/dags/data_exports/gobi_transmission.py) is receiving the gobi list and it probably needs to be a dict instead of a list:

 transmit_data = transmit_data_ftp_task(
        "ftp-ftp.ybp.com-stanford", generate_isbn_list
    )

Changing gobi_list_from_marc_files like this will make the dag error go away, but there are likely downstream, changes that need to be made to the transmit_data_ftp_task.

def gobi_list_from_marc_files(marc_file_list: str):
    gobi_lists = { "file_list": [] }
    gobi_transformer = GobiTransformer()
    for marc_file in ast.literal_eval(marc_file_list):
        gobi_lists["file_list"].append(gobi_transformer.generate_list(marc_file=marc_file))

    return gobi_lists
jgreben commented 2 months ago

Relates to https://github.com/sul-dlss/libsys-airflow/issues/986