GOBI selection DAG failing on transform

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows

Apache License 2.0

5 stars 0 forks source link

GOBI selection DAG failing on transform #1026

Closed ahafele closed 3 months ago

ahafele commented 3 months ago

https://sul-libsys-airflow-dev.stanford.edu/dags/select_gobi_records/grid?run_id=manual__2024-05-24T17%3A59%3A43%2B00%3A00&execution_date=2024-05-24+17%3A59%3A43%2B00%3A00&tab=logs&dag_run_id=manual__2024-05-24T17%3A59%3A43%2B00%3A00&task_id=transform_folio_marc_record

jgreben commented 3 months ago

Investigation reveals that all of the gobi marc records are being excluded because of this code block in the marc exporter. Can you or @jermnelson clarify the requirements for excluding the output of marc records depending on the vendor? The gobi transformer also checks the 035 in order to determine whether lines should be excluded from the gobi list, but it never gets that far because of the exclude_marc_by_vendor function in the upstream task.

ahafele commented 3 months ago

I assume that block is meant to address the requirements of

Exclude records with gls in the 035
Exclude if its 008 field has a language code other than eng or fre

But I don't think it is working correctly if the result if 0 records.

jgreben commented 3 months ago

See #1030

ahafele commented 3 months ago

@jgreben I tried to retest - the DAG is successful but no new files are generated. Do the old ones need to be removed first?

jgreben commented 3 months ago

Yes, I did not activate the gobi transmission dag so it was just appending to the previous generated file. I activated it so now hopefully it will work as intended.

ahafele commented 3 months ago

Same result. @jgreben could you delete all the gobi files and I'll try with a fresh start?

ahafele commented 3 months ago

Working now.