Closed constanca-m closed 5 months ago
I believe we are doing the same thing for elastic agent: https://github.com/elastic/elastic-agent/blob/b7617cf47f1acd0e77022e7817e2103c347af917/internal/pkg/composable/providers/kubernetes/pod.go#L60 and creating multiple watchers there, instead of sharing. Should we create a new issue for that @MichaelKatsoulis ?
I believe we are doing the same thing for elastic agent: https://github.com/elastic/elastic-agent/blob/b7617cf47f1acd0e77022e7817e2103c347af917/internal/pkg/composable/providers/kubernetes/pod.go#L60 and creating multiple watchers there, instead of sharing. Should we create a new issue for that @MichaelKatsoulis ?
This is the kubernetes provider. The provider like the add_kubernetes_metadata processor start watchers but only 3. One for pod, one for node and one for namespace. I mean that they are not per metricset.
Background
This issue was first detected when trying to stop node and namespace watchers from starting (issue is this one). At first, it seemed that we only had these watchers based on features
add_resource_metadata
andhints
, but having a better look, we were actually starting watchers at many places.Issue
All
state_*
metricsets are using this function https://github.com/elastic/beats/blob/d9139c95a0b0bd74c09949a3265a0959e2d43e9c/metricbeat/helper/kubernetes/state_metricset.go#L46 to start, except fromstate_container
andstate_resourcequota
.This function adds a metadata enricher to each metricset by calling
NewResourceMetadataEnricher
function: https://github.com/elastic/beats/blob/d9139c95a0b0bd74c09949a3265a0959e2d43e9c/metricbeat/helper/kubernetes/state_metricset.go#L91Also pod and node metricsets call NewResourceMetadataEnricher to enrich the events with metadata.
This enricher will create 3 watchers: https://github.com/elastic/beats/blob/d9139c95a0b0bd74c09949a3265a0959e2d43e9c/metricbeat/module/kubernetes/util/kubernetes.go#L158
One for the resource of that metricset, one for node and one for namespace.
Since we can have multiple metricsets enabled, then we will also have multiple node and namespace watchers running. One for each metricset.
To solve this, we need to find a way to share these watchers between the metricsets, similar to what we did to fetch metrics from KSM (PR) and Kubelet.
Additionally, from the way we create the resource specific watcher https://github.com/elastic/beats/blob/d9139c95a0b0bd74c09949a3265a0959e2d43e9c/metricbeat/module/kubernetes/util/kubernetes.go#L454 we can see that in case the function was called from
state_node
/state_namespace
metricset (resource is node/namespace), we will end up creating yet another watcher for node/namespace. We need to, at least, add a condition to stop the duplicated watcher from starting.More on this, we seem to always start the three watchers for all the resources that the
NewResourceMetadataEnricher
is called by. These watchers are created for metadata enrichment and for some resources the existence of one or some of these watchers is not relevant. For example node watcher is not needed in case we want to enrich deployments or statefulsets. We need to start only the relevant watchers for each resource.Possibly this can be solved if we share the watchers between metricsets.
The better handling of watchers initialization will lead to less Kubernetes API calls and possible issues related to that in large scale clusters.
Issues
Relates to https://github.com/elastic/elastic-agent/issues/3801.
Current watchers
These are the current watchers when starting metricbeat with default configurations.
For example, like this.
``` - type: kubernetes scope: cluster node: ${NODE_NAME} unique: true templates: - config: - module: kubernetes hosts: ["kube-state-metrics:8080"] period: 10s metricsets: - state_node - state_deployment - state_daemonset - state_replicaset - state_pod - state_container - state_cronjob - state_job - state_resourcequota - state_statefulset - state_service - state_persistentvolume - state_persistentvolumeclaim - state_storageclass - state_namespace ```For the
state_*
metricsets grouped as described above:For all the other metricsets:
Expected watchers
Watchers needed for each metricset by default (without counting
add_resource_metadata.deployment/cronjob
):Checks
PRs