[processor/k8sattributes] Retrieve pod metadata from kubelet

[bacherfl]

Description: This is a proof of concept for the k8sattributes processor to retrieve pod metadata from the kubelet instead of the Kubernetes API server. Note that this is just to illustrate how a potential implementation could look like, and what the limitations are to have a base for a discussion on how to go forward with the linked issue.

So far, the most significant implications of using the kubelet instead of the Kubernetes API server are the following:

• The kubelet API is not officially documented and potentially prone to change. The best source of documentation in this case is the code of the kubelet
• The Kubernetes client-go library does not provide support for directly interacting with the kubelet API - There is the possibility to reach the kubelet's endpoints via the client's WrapTransport option where the request URL can be set to the /pod endpoint of the kubelet, so the authentication configuration of the kubernetes client can be reused, but this is a bit of a hack, and comes with several limitations, as outlined in the following points
• The kubelet API only provides a really simple endpoint for retrieving all pods (or only the running pods) on the node. The structure of the response is the same as for the pods list endpoint of the kubernetes API, but the kubelet endpoint does not provide any filtering capabilities. So the Label/Namespace/Attribute filters will all have to be applied in the collector after retrieving the list of Pods
• No support for using Informers. The current implementation relies on the use of informers that call the Watch function of the client to receive an event every time one of the observed resources is added/updated/deleted (e.g. when a pod is added or deleted). This is not possible with the kubelet API, so we'd have to rely on polling the /pod endpoint regularly, and synchronise the received list of pods with the latest known status of the collector. This comes with quite an overhead if we want to maintain the Added|Updated|DeletedPods counters which are exposed as metrics by the collector as we have to compare the retrieved pods against the list of pods the collector currently has in memory to know if any pods have been deleted or added (see the diff of the PR).

Link to tracking Issue: #14475

Testing: Only manual tests right now, as this is still only a draft PoC

Documentation:

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

Closed as inactive. Feel free to reopen if this PR is still being worked on.