Implement "insert" or "combine" case for adding new data

[rt17603]

What is your issue?

See #591 for original details on this. At the standardisation layer (standarise_* functions) we want to ensure that the user can specify how to add / combine new data with currently existing data in the object store.

For the new if_exists input flag (within a standardise_* function) we want to implement a case for an "insert" option.

• Previous data source, new data added, overlapping date range / replacement date range
  • if_exists="auto" (default)
    • Proposed: Returns helpful error, don't add (will be added through PR #831 )
  • if_exists="new" flag
    • Proposed: "Retire" ALL previous data to older version, create new stack just with new data (new version) (will be added through PR #831 )
  • if_exists="insert" flag / if_exists="combine" flag
    • Proposed: Replace overlapping date range - delete old, insert new (keep current version, "Retire" version with previous data but create new version with combination of "old" and "new" data).
      • Caveat: need to make sure NaN values and data just not being present is treated differently. NaN should be replaced and then dropped rather than just ignored. This may need to be addressed at the parse_* function level.

[rt17603]

PR #1166 is working to address this issue, including laying groundwork to support this.

For this the suggestion is to approach this with 3 goals in mind that build up this functionality:

1. Update version handling (neutral, restructure change)
   • Would be useful to include some diagrams of how the classes (e.g. ABC, Store, VersionedStore, LocalZarrStore etc. build on and relate to each other)
2. Formalise and update index comparison (may be neutral or may include updated functionality)
3. Use these new structures to add the new mode for combining data