Open Jeansen opened 2 months ago
@Jeansen the collector CR (the first code snippet) should have mode set to sidecar
@pavolloffay Oh, sorry. That is a typ or better copy/paste error after I had tried different settings. I updated the code accordingly.
@frzifus can you help here? IIRC you looked at core k8s components, did you try to inject sidecar there?
yes, I think it depends a bit on how you setup your k8s cluster. If for example your api-server and another component would run with hostNetwork: true
there would be a conflict with the sidecars.
Instead we deploy a daemonset running with hostNetwork: true
. That way, etcd, kubelet and the apiserver can report to the collector on each node.
Having a sperate CR could make sense. But tbh I did not think much about it.
Could you provide an example of what you would like to see @Jeansen ?
Kubecon tutorial [link].
FTR:
@frzifus Well, regarding e.g. the apiserver, I'd expect it to behave the same way as any other service when using sidecars. But as I wrote, it does not with respect to static Pods. So I tried using the Deployment mode. First, it did not work either because static Pods cannot use the cluster DNS. I reached out to the Kubernetes architects and they wrote:
Control plane components run with host networking; because of what these components do, they need to be able to boot before there is a healthy control plane for your cluster.
So, I created an ingress for the collector and with an external DNS it now works, at least for me (without sidecars).
Her's what my config looks like for the apiserver:
apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
#endpoint: "10.98.30.28:4317"
endpoint: "otel.k8s.lab:80"
samplingRatePerMillion: 1000000
Of course, as seen above in the comment, one could also use a specific IP of the collector service, but that's hardly what we want!
And here's the Ingress manifest:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
name: otel-ingress
namespace: kube-system
spec:
ingressClassName: nginx
rules:
- host: otel.k8s.lab
http:
paths:
- backend:
service:
name: otel-kube-system-collector
port:
number: 4317
path: /
pathType: Prefix
Finally, the apiserver needs to have --tracing-config-file
set accordingly.
That is my current solution on my "way to Rome". If sidecars would work, all this would not be necessary. Even more, the TracingConfiguration
would be quite generic using localhost
instead of a fixed DNS name.
Component(s)
collector, auto-instrumentation
What happened?
Description
No sidecar Containers created for static pods, e.g. kube-apiserver.
Steps to Reproduce
Create a custom resource, with mode set to sidecar in namespace kube-system. Add relevant annotation to this namespace.
Here's my current playground CR:
And here's what my kube-system NS looks like:
Expected Result
When deleting a kube-system pod, e.g. kube-apiserver, the new Pod should have a sidecar container.
Actual Result
There is no sidecar container. BUT: If I create a simple dummy deployment in the kube-system namespace, then a sidecar container will be created.
Kubernetes Version
1.30.0
Operator version
0.56.0
Collector version
0.56.0
Environment information
Environment
Nodes run on Debian 12 as VMs (QEMU/KVM). I have 3 masters, 4 workers. Container runtime is CRI-O 1.30.0
Log output
No response
Additional context
I am following along https://kubernetes.io/docs/concepts/cluster-administration/system-traces/ but without the sidecar, I had to resort to a deployment.