NASA-PDS / registry-sweepers

Scripts that run regularly on the registry database, to clean and consolidate information
Apache License 2.0
0 stars 1 forks source link

Deferred partial updates #116

Closed alexdunnjpl closed 2 months ago

alexdunnjpl commented 2 months ago

🗒️ Summary

Implements handling of case where non-aggregate product histories - which are generated by stateless iteration over registry-refs collection documents as implemented in #115 - are shared among collections having a different LID than the one nominally indicated by the first five chunks of the non-agg's LID.

If multiple updates are detected for a single nonaggegate product, all records subsequent to the first are shelved to disk, and are later merged with each other and the first, which is retrieved from the database (as it has already been written - this is ensured by a write buffer flush prior before processing of deferred records). After merging, the full history of the deferred nonaggregate products is yielded and written to the database.

⚙️ Test Data and/or Report

Functional tests updated and pass. Manual testing in progress.

♻️ Related Issues

Fixes #114