Nonaggregate products present in "foreign" collections/bundles do not have correct ancestry.

[alexdunnjpl]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

As previously noted by @sjoshi-jpl's log error reports, there have been numerous cases of "Multiple updates detected for doc_id [...]"

I've tracked this down to what appears to be an issue where nonaggregate products who belong to collections/bundles not indicated by the non-aggregate lidvid produce multiple disjoint ancestry histories.

This will need to be accounted for with ad-hoc behaviour, possibly caching the colliding updates until after the next buffer flush (as the collision may not have been written to db at detection-time), then performing read/merge/write for each affected document, with the missing content.

🕵️ Expected behavior

I expect registry-sweepers to run (tested against psa-prod) without the production of "Multiple updates" errors

📜 To Reproduce

Run sweepers against psa-prod

⚙️ Engineering Details

This makes sense, given the fact that it's iteration over registry-refs docs (i.e. pages of collection-specific non-agg references) which produces the non-agg records. This is compounded by the new iterate-by-collection-lid approach.

Another way to phrase the issue:

• we have to process the document corpus in chunks because it's too large to keep in memory
• each chunk produces ancestry records (here, a single record is the collection and bundle membership for a given product) which get written to the products' docs in the db
• in the corner case, a product appears in a couple of different chunks, resulting in multiple partially-complete records which, if just written naively, result in the product doc only containing the ancestry content of the latest partial history to hit the db so the solution (the only reasonable one I can think up) is to detect when this happens, and address it by adding a "read, merge, write" step in that case rather than just writing each partial record over the top of the prior one

[alexdunnjpl]

Example:

{
  "lidvid": "urn:esa:psa:context:instrument_host:spacecraft.tgo::1.1",
  "parent_collection_lidvids": [
    "urn:esa:psa:context:instrument_host::1.0"
  ],
  "parent_bundle_lidvids": [
    "urn:esa:psa:context::1.0",
    "urn:esa:psa:context::2.0",
    "urn:esa:psa:context::3.0",
    "urn:esa:psa:context::4.0"
  ]
}

{
  "lidvid": "urn:esa:psa:context:instrument_host:spacecraft.tgo::1.1",
  "parent_collection_lidvids": [
    "urn:esa:psa:em16:context::1.1"
  ],
  "parent_bundle_lidvids": [
    "urn:esa:psa:em16::1.0",
    "urn:esa:psa:em16::10.0",
    "urn:esa:psa:em16::100.0",
    "urn:esa:psa:em16::100.1",
[...]
    "urn:esa:psa:em16_tgo_acs::1.4",
    "urn:esa:psa:em16_tgo_cas::1.3",
    "urn:esa:psa:em16_tgo_nmd::1.2"
  ]
}

[alexdunnjpl]

Prioritised and added to current sprint, per @jordanpadams @tloubrieu-jpl

[alexdunnjpl]

Status: in-prog, need to

• do a little work on ironing out a couple of functional tests
• implement the write buffer flush between yielding non-conflicting records and merging deferred records.

[alexdunnjpl]

Status: complete, PR opened

[alexdunnjpl]

Resolved by #116