elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
42 stars 451 forks source link

[Kubernetes]: [Metrics] Kubernetes Overview dashboard aggregation leads to max_buckets exceed #11464

Open Alphayeeeet opened 1 month ago

Alphayeeeet commented 1 month ago

Integration Name

Kubernetes [kubernetes]

Dataset Name

No response

Integration Version

1.68.1

Agent Version

8.15.2

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.2

OS Version and Architecture

RHEL 8.9

Software/API Version

No response

Error Message

Visualization: error while executing search Control element: Internal Server Error (at least sometimes, as I couldn't reproduce for screenshot).

The Elasticsearch log prints out this message: Image

Event Original

No response

What did you do?

After some of the previous integration updates (presumably https://github.com/elastic/integrations/pull/10593), the newly added visualizations exceed the bucket limit, and break also the filter control element (sometimes, as I couldn't reproduce when creating the screenshot.). The control element for "Cluster Name" then said: "Internal Server Error.

What did you see?

A uncompletely loaded dashboard like this: Image

What did you expect to see?

A fully loaded dashboard like this:

Image

Anything else?

Important side node: The time range in all the above screenshots is past 15 min. Longer timeranges not even considered.

Also: Increasing the max_bucket setting helped. But only after increasing it excessively to > 100k and < 1 mio (I didn't try any in between).

Alphayeeeet commented 1 month ago

Suggesting also somehow, to set a different data view in the built-in dashboards, as metrics-* can be quite large in a fully built production environment. This could be a huge performance improvement for future, so please consider creating a separate issue in the affected projects for this.

Also as an alternative: Consider integrations to create data-views for them, that query only the respecitive module like in this case metrics-kubernetes.*