[Logs Overview] Enhanced logs component for solution UIs

[weltenwort]

📓 Summary

Instead of just showing all log entries in the embedded logs stream, we want to summarize the log entries and surface the most important categories to the user. To that end we want to create an component that can be used in solution UIs to offer an enhanced view of the log entries specific to that context.

• :link: part of: https://github.com/elastic/observability-dev/issues/3660

✔ Acceptance criteria

• These features are implemented in a shared component that can be embedded into solution apps and Discover.
• The components takes the following inputs:
  • indices, optional string, defaults to the logs advanced setting
  • time range, optional iso date strings
• If no time range is given the component renders its own date range picker. Otherwise it uses the given time range.
• The component categorizes the log messages using the categorize_text aggregation and runs a change_point aggregation for each. To ensure it performs well with large amounts of log entries it employs a two-stage process if a threshold has been exceeded:
  • the first stage wraps the aggregation into a random_sampler with a dynamically detected sampling rate and puts a lower limit on the category size
  • the second stage performs the same categorization, but without sampling or limits and with the categories of the first pass excluded
• The component shows a data grid of the categories with the following columns, the exact names of which are still to be discussed:
  • Pattern
  • Event count
  • Event type
  • Change point
  • Histogram
• The pattern column shows the terms of the category concatenated with " * " as a joining string. The terms and the joining string are formatted in a clearly distinguishable way.
• The event type column shows a label for the type of change detected, which is derived from the change_point agg result and the histograms before and after the change point timestamp.
• The change point column shows a timestamp of the change, which is either derived from the change_point agg if it yields any results, or the first occurrence of a rare event for which the change_point failed. Steady state categories don't have a change point timestamp.
• The histogram column shows a histogram chart of the document counts of each category. If a change point timestamp exists, it is annotated on the chart.
• The rows can be sorted by count, type and change point. The order of the types is such that "rare" categories are sorted highest and "steady state" categories are sorted lowest.
• There is a "open in Discover" link that navigates to Discover while preserving the indices and time range.
• If not categories can be extracted a suitable empty state is being shown.
• A new feature flag is introduced to control the usage of the new component. The component is only used when the user has turned on the feature flag.

:art: Mock-ups

:warning: These mock-ups are not authoritative and are only meant to guide the implementation.

💡 Implementation hints

• A PoC with the two-pass sampling technique can be found in https://github.com/weltenwort/kibana/tree/poc-enhanced-logs-embeddable.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)