Mapping before indexing comparison doesn't look at nested fields

[ajakz]

Summary

Prior to indexing in the Search and A&E indices, the fields for the existing ES mapping are compared to the available content. Only fields that are in the mapping are indexed and excess fields are dropped prior to sending to ES. This keeps ES from throwing errors on indexing but doesn't work on objects/nested fields.

Tasks

• [ ] ~Update comparison logic to look at nested/object fields~
• [ ] ~Add warning logging for fields that are dropped~

Updated tasks:

• [x] On Indexer startup, compare any existing mapping with expected mapping
  • WARN if fields added or deleted but continue to startup
  • ERROR and kill app if any existing field type or analyzer changes
• [x] Remove field checking from TransformationUtils
• [x] Add explicit building of all index types
• [x] Document reindexing strategy steps/best practices for ops when mappings change

[ajakz]

Updated tasks:

• [ ] On Indexer startup, compare any existing mapping with expected mapping
  • WARN if fields added or deleted but continue to startup
  • ERROR and kill app if any existing field type or analyzer changes
• [x] Remove field checking from TransformationUtils
• [x] Add explicit building of 4 index types
• [ ] Document reindexing strategy steps/best practices for ops when mappings change

[zebdelk]

Helping with testing

[dneufeldcu]

Throw error when application mappings don't match ES types.

[dneufeldcu]

Zeb is working through manual test process and documentation of the error thrown.

[dneufeldcu]

@zebdelk Please update this story with references to code changes and documentation to date

[dneufeldcu]

See #1360

[erinreeves]

Remove field checking from TransformationUtils accomplished by other PR by using POJO's

[erinreeves]

Manual notes: basically, start the app (use kibana or similar to verify mappings), then go make those or similar kinds of edits to the mappings in elastic-common/src/main/resources/mappings, then restart app (or let skaffold do it depending on how you run it)

[erinreeves]

It gives me some pause that it is so hard to test this one scenario, but it's so hard to inject our own mapping files. It's always been my feeling, and I've read a few places, that if something is difficult/impossible to test that can be a sign the code should be changed. That's a really tight coupling. I was thinking you could make a backup and copy your own version to those mapping files but I'm unsure how long to wait or know when they have been reloaded to have the test then continue. It feels like a lot of work for such a tiny thing.

Chatted with Zeb and concluded to do this right would be almost more confusing.