Supporting custom index patterns for Trace Analytics - Data Prepper

[derek-ho]

• Add support for Jaeger custom indices
• Add support for Data Prepper custom indices

Additional option/indices shown by detecting mapping on indices

[derek-ho]

This will allow users to select a specific index that may correlate to a specific naming convention for a given app/subset of users in trace analytics and app analytics.

Feature Proposal [WIP]

• Trace Analytics

Currently Trace Analytics has a data type selector - Data Prepper or Jaeger, I am proposing to add 2 sub-selector under Data Prepper where users will be able to select a specific index/indices/index pattern on which they will be able to search for traces - one for the span index and one for the service map index

We will first grab all of the indices through the cat indices API, get all of those indices (excluding indices that start with a dot(.))and their mappings through the indices get mappings API, and add a constant representing the data prepper index mapping format. We will filter out any indices that do not follow that index mapping format, and allow users to select index/indices/index pattern on the resulting list. (Are there any security concern? Any network bandwidth limit exceeded concern over getting the mappings of all of the indices?)

[derek-ho]

Had a conversation with @YANG-DB, this will not be easily supportable via checking mappings - should be done via sampling/other means. High level goal is: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html. Moving this out of 2.6 tasks