New DAG for replicating UpdatingMarcToInstanceMapping.sh BASH Script

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows

Apache License 2.0

5 stars 0 forks source link

New DAG for replicating UpdatingMarcToInstanceMapping.sh BASH Script #842

Open jermnelson opened 5 months ago

jermnelson commented 5 months ago

Part of the upgrade to Poppy, we need to update all of the MARC to Instance Mappings that replicates the functionality in UpdateMarcToInstanceMapping.sh but also allowing for this process to be parallelized as the current estimate to run the bash script is a few weeks given our ~11 million Instance/SRS records.

This DAG will be usefully for future workflows when the MARC mapping changes and the Instances need to be updated in collections.

jermnelson commented 1 month ago

There are two new Okapi endpoints that may do this re-mapping without us needing to run a DAG to re-map the MARC records to Instances:

Reindex instances , open question is if a re-index of the instances does this re-mapping?
MARC Migrations this endpoint has two parameters, entityType could be set to "instance", and operationType set to "remapping"

We should test these two options to see if it works as expected and look for any performance impacts in our K8s clusters.