Enable amending initially configured locations

[rikukissa]

Description

So in other words, make it possible to correct existing details. In this task we build the foundation for temporality of Locations.

A location is created in the wrong hierarchy during initial configuration or with the wrong spelling. Moving the location hierarchy for the location or updating the spelling should update all records associated with it be they “declared, validated (e.g. active workqueue)” or “issued” and therfore no longer in a workqueue.

Acceptance criteria

• It is now possible to change location's name or parent location after deployment of OpenCRVS
• If I change the name of a location, two things should happen:
  • All places where this location is referred to have the new name
• If I change the parent location of an office
  • In advanced search, records can now be found when searched with the new parent location
  • In advanced search, records can not be found when searched with the old parent location

Notice that in this first phase, for location information queries to Hearth, we can still heavily rely on the Location / Location_history functionality and none of the MongoDB queries for fetching location details should change. If a location name or partOf is changed, the previous version gets pushed to Location_history and we don't have to worry about it too much.

Known limitations

• Metabase is outside the scope
• Administrative structure changes (additions, modifications, removal) cannot be added pre-emptively

Technical tasks

This model changes location details directly and does not provide "temporality". If you change location name today, even records from 10 years ago show the new location name. Same with if you move the location to a new parent location.

• [x] Build an encapsulated "office / leaf level resolver" endpoint to the config service. You can supply a location id to the endpoint and it returns a list of office ids and validity periods under the location. (locationId: LocationID) => Array<{locationId: LocationID, validFrom: Date, validUntil: Date}>
  • validFrom can be hard-coded to 1/1/1970
  • validUntil can be hard-coded to 1/1/3000
• [x] Refactor full paths to be leaf references in ES, PractitionerRole and Users (catchmentAreaIds)
  • [x] Check if catchmentAreaIds is used anymore ( gateway resolver needs to be updated)
  • [x] practitioner & practionerRole will need migration as catchmentAreadIds are stored in there
  • [x] Performance and record searches should now be changed a little to be based on the offices and validity periods gotten from the office resolvers.
  • We currently only write to in_complete_fields so no need to change any queries
  • [x] Build a way to migrate ES to the leaf level id's
  • [x] Write tests for the indexing functions in search now that they're immutable, so it should be easier to test.
• [x] Fix the PR which updates the Postman collections https://github.com/opencrvs/opencrvs-countryconfig/pull/115
• [x] Refactor full paths to be leaf references in InfluxDB. Research what needs to be done in metrics/performance to work with the leaf level changes. It might be nearly nothing.
• [x] Update 1.6.0 changelog to include the reindexing changes so that upgraders understand the ES indexing has changed
• [x] Remove eventJurisdictionIds from search; they are not referenced in very many places
• [ ] Update the sequence diagram markdown files in places where location is being searched. Also a good time to double-check where locations are searched doing multiple calls at the same place to refactor them to use the hierarchy endpoint.
• [ ] #7468
• [ ] #7467
  • Fetches a list of locations
  • If any item of location list has changed, (so one of the following fields change name, partOf):
  • Update the location with details given by country config
  • If items have been removed: invalidate the existing location by changing its status to inactive

[rikukissa]

CSV changes in country config works like this

1. Version in CSV (or country config API)

JWMRGwDBXK,Ibombo District Office,Location/oEBf29y8JP8,CRVS_OFFICE

So seeding created this

name: Ibombo District Office
valid_from: 1970
valid_to: 27/3/2024
is_valid: true 

is_valid flag is only for demonstrative purposes.

2. Version in CSV (or country config API) when a new deployment was made 27/3/2024

JWMRGwDBXK,Ibombo's District Office,Location/oEBf29y8JP8,CRVS_OFFICE

The existing record was changed to

name: Ibombo District Office
valid_from: 1970
valid_to: 27/3/2024
is_valid: false 

New record was written

name: Ibombo's District Office
valid_from: 27/3/2024
valid_to: 1/1/3000
is_valid: true

3. Version in CSV (or country config API). Validity period now defined

JWMRGwDBXK,Ibombo's Great District Office,Location/oEBf29y8JP8,CRVS_OFFICE,1/5/2024,1/1/3000

The existing record was changed to

name: Ibombo's District Office
valid_from: 27/3/2024
valid_to: 1/1/3000
is_valid: false 

New record was written

name: Ibombo's Great District Office
valid_from: 1/5/2024
valid_to: 1/1/3000
is_valid: true

[naftis]

• [ ] Add an example script to country config service for updating locations: 2 days
  • Fetches a list of locations
  • If any item of location list has changed, (so one of the following fields change name, partOf):
  • Update the location with details given by country config
  • If items have been removed: invalidate the existing location by changing its status to inactive

@Zangetsu101 @rikukissa This could make sense to be within core in data-seeder. A yarn seed:locations script similar to the current location seeding. data-seeder already has the means to fetch the locations from country-config and while it does stuff via the temporary super user, this seeding could require setting a token

[Zangetsu101]

The issue with the super user is that we deactivate it as soon as seeding's finished

[naftis]

The groundwork for allowing to change location structures is done. Now we only save the leaf level locations and the full path of a location is then fetched from ground up.

For the location configuration to be a documented and configurable feature, we depend on a real need from a country/installation. There are options for how to allow configuring the locations (UI, or only a Postman example request, or a GitHub Action which re-seeds the locations...), but we are missing a real-world use case to finish the design of the feature

[naftis]

Remove eventJurisdictionIds from search; they are not referenced in very many places

1. the event is in health location, we use eventLocationId
2. if the event isn't in a health location, the full path is still saved.
3. if the event is international, whatever is inputted to the fields is being saved. We do not allow searching international events with other than country though, so that can safely be removed.

^{ElasticSearch rows}

Case 2 needs to be refactored when the actual UI/location editing feature is created. I'd say it's a small issue though, as the searching itself already uses leaf level locations, and we simply can then reindex the ElasticSearch to only save the leaf level.