search

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows
Apache License 2.0
5 stars 0 forks source link

Logging for full dump appears to be broken #1071

Closed shelleydoljack closed 5 months ago

shelleydoljack commented 5 months ago

There is logging to let us know which file an instance ID was written to: https://github.com/sul-dlss/libsys-airflow/blob/b25279c821086ae57c7114d1c120bae09522481b/libsys_airflow/plugins/data_exports/marc/exporter.py#L134-L136

However, on the latest DAG run of select_all_records, the logs are missing for each mapped task in fetch_folio_records: Screenshot 2024-06-11 at 2 49 32 PM Screenshot 2024-06-11 at 2 49 59 PM

https://sul-libsys-airflow-dev.stanford.edu/dags/select_all_records/grid?dag_run_id=manual__2024-06-04T12%3A35%3A33-07%3A00&tab=mapped_tasks&task_id=fetch_folio_records&map_index=0

jermnelson commented 5 months ago

This isn't a "bug" in the code but how we deploy the application using capistrano. Each time we deploy a branch, the logs directory is re-created as an empty directory while the previous version does retain the logs.

shelleydoljack commented 5 months ago

Oh right, of course. I guess we should be more weary of that if we want to go back and try to track where a record from the full dump might have been written.