search

sul-dlss / dlme

Digital Library of the Middle East web application, based on Spotlight
https://dlmenetwork.org/
Other
20 stars 2 forks source link

Need a solution for automatically deleting bulk tags for a given collection from prod, uat #1581

Closed jacobthill closed 1 year ago

jacobthill commented 2 years ago

This ticket is related to a larger process of automatically syncing metadata in DLME. Another part of this process captured in https://github.com/sul-dlss/dlme/issues/1580.

The most recent Infrastructure work cycle (September 2022) focused on completely automating the ETL pipeline for most data providers. One lingering issue preventing full automation relates to the way bulk tags work in Spotlight and our existing curator workflow. Bulk tags are manually assigned to records by DLME curators in the dev instance of the web app. These tags persist when metadata is refreshed. If a curator assigns tags in dev, exports them, imports them into prod, then deletes some of them in dev, exports them, and imports them again to prod, the tags that were intentionally deleted in dev will not be deleted in prod (I think). To solve this problem a curator has to manually delete all bulk tags for a collection from prod before importing them from dev. This is currently an obstacle to automation. Ideally we need a way to delete all tags from a collection of records from Airflow so that this deletion can be part of the Airflow automation and https://github.com/sul-dlss/dlme/issues/1580 can be implemented safely.

aaron-collier commented 2 years ago

One option may be that when applying the tags from a bulk update, the existing tags for the individual items are removed before applying the new tags.

jacobthill commented 2 years ago

That could work as long as the tags never get deleted from dev. In this case dev is the source of truth for the curator tags, which is not ideal since its a dev environment and things are more likely to go wrong in dev. I am open to ideas on a better way of doing this. Would it make more sense to do curation in uat and sync tags to dev and prod? My main concern is that the manual work is never lost because something we did inadvertently deleted the tags.

aaron-collier commented 2 years ago

yes, apologies if that wasn't clear. Maintain dev as the source of truth, but on import to prod/uat, the individual items tags are removed before the tags from the import are applied. Assuming no curation happens in those environments, and only happens in dev, this should be fine.

jcoyne commented 1 year ago

Solved because workflow changes to happen completely in production when we move on prem.