Feature request: Tool to fix mapping conflicts

[ThomasdOtreppe]

From time to time, due to testing with Logstash filters, mapping conflicts happen and Kibana is pretty good at telling which fields have a conflicts but that's pretty much it.

Finding which indices are affected is more or less easy (Is there a query to find field with a specific name and specific type such as integer, string?) and takes a bit of time, then there is the reindexing part that is not easy.

Due to the complexity of the procedure, besides fixing the issue in Logstash, I pretty much give up when it happens and wait for the indices to be curated.

It would be nice to have a tool in Kibana to fix those mapping conflicts. Here is how I think (simplified) it could look like: Display all the fields that have conflicts and for each of them, allow to convert to a specific type. Once all of them have been selected, just click 'Apply' to fix it.

[rashidkpc]

Waiting for this: https://github.com/elastic/elasticsearch/issues/15201

[aut0maticjack]

+1 also, https://github.com/elastic/elasticsearch/issues/15201 is closed and stable.

[alefebvre-ls]

👍

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[cjcenizal]

I'm adding the ES UI team to this issue because we're currently working on a mappings editor (https://github.com/elastic/kibana/pull/47562) and we have a reindex wizard on our roadmap (https://github.com/elastic/kibana/issues/46755). We may actually be the best owners for this.

[PhaedrusTheGreek]

Having an intuitive & simple way to fix mapping conflicts would be of great service to stack users. There's often nothing we can do about incoming garbage data, but the trick is to prevent the garbage from making the mappings, and to index: false or ignore_malformed: true instead.

Here's a proposed workflow:

(1) - Identify mapping conflicts:    i. Monitoring App can identify real-time errors in Elasticsearch Logs    ii. Management App shows Index Pattern conflicts (2) - Identify existing matching template(s) and ask user to select (or create new) (3) - Open Mapping Editor UI and auto-edit the given field, allowing user to select data type    i. potentially smart-suggest a default on existing indices majority for this field.
   ii. Also suggest ignore_malformed here    iii. Suggest index: false (correct choice for garbage data) (4) - If current index is non-compatible, offer immediate ILM rollover (5) - List non-compatible indices and offer to Reindex

The "compatibility" should be determined only in the context of the field in question, otherwise it will probably always appear non-compatible

Additionally the same workflow could solve other problems such as indexing failed due to field length too long , requiring ignore_above to be set, etc..

[PhaedrusTheGreek]

Related: https://github.com/elastic/elasticsearch/issues/51688

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)