[Index Management][Serverless] Extend Index Management UI to visualize the data usage over the time

[YulNaumenko]

Serverless users should be able to analyze how much data they are ingesting (daily/weekly/etc.) and retaining over the selected period of time. Current Kibana Index Management page shows only the storage size per datastream at the current point of time. The goal of that issue is to describe the UI scope for the datastreams usage visualization. The Kibana API which will serve it is described as a separate ticket https://github.com/elastic/kibana/issues/192965

Extend Data section cards with the new link to the Data Usage page:

The page components:

• 2 custom Time series charts show the ingested data and retained data over a period of time (with ability to switch between line and bar)
• time range filter to select from and to dates
• chart should has the legend which reflects the displayed datastreams and could work as a filter to show/hide the data per datastream
• control to manage which time scale to use - hour/day/week/month

All the interactive components will call the internal Kibana API here https://github.com/elastic/kibana/issues/192965

Mockups: /// TODO

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[MikePaquette]

Based on several inquiries today, I think we can focus our design work by prioritizing the classes of use cases we want to support with this visualization.

We can classify these use cases into three buckets as summarized below. There may be more.

I propose that we prioritize bucket 1, "Find Heavy Hitters" because this bucket is most closely related the usage and billing transparency we seek to provide to our serverless customers. Buckets 2 and 3 are indeed useful, but if we treat them as secondary priorities, we may be able to make good design decisions quickly.

1. Find Heavy Hitters - determine the Top N data streams that are ingesting at highest rates and/or retaining the highest data volumes to answer questions like:

   • Heya data engineer, your project monthly charge is getting pretty high, can you trim down your retention settings, and get rid of some old data?
   • Heya analytics-person, your dashboard is really slow, how much data is in those indices that you’re querying?
   • Heya platform engineer, what’s your average daily ingest volume, in GB/Day for all your windows data streams?

2. View Ingest rate patterns to answer questions like:

   • does this/these/all/any data streams have a periodicity, perhaps day/night or weekday/weekend
   • What’s the current ingest rate right now for this/these/all data streams

3. View Ingest rate anomalies to answer questions like

   • Are we ingesting more EDR logs today than we were at this time last week?
   • Is there an unusually high rate of ingest on this/these/any/all data streams
   • Is there an unusually low rate of ingest on this/these/any/all data streams

[neptunian]

Thanks @MikePaquette I think this would be useful for design as I'm not sure these issues are addressed and the design is still in flux. This design is outdated and I think the new one is still being worked on.

[ashokaditya]

https://github.com/elastic/kibana/pull/193966 https://github.com/elastic/kibana/pull/195556 https://github.com/elastic/kibana/pull/196559

[ashokaditya]

https://github.com/elastic/kibana/pull/197056

[ashokaditya]

Test PR https://github.com/elastic/kibana/pull/198007