auto instrumentation not working in eks 1.27

shishir84 commented 9 months ago

Describe the bug auto instrumentation not working in eks 1.27

Steps to reproduce kubectl logs instrumentation.opentelemetry.io/auto-instrumentation -n error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "pkg/scheme/scheme.go:28"

What did you expect to see? kubectl logs instrumentation.opentelemetry.io/auto-instrumentation -n should give me logs.

What did you see instead? error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "pkg/scheme/scheme.go:28"

What version did you use? image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.90.0 - manager image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 - rbac

What config did you use? apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: auto-instrumentation spec: exporter: endpoint: "http://observability-collector.aws-otel-eks.svc.cluster.local:4317" propagators:

tracecontext
baggage
b3 sampler: type: parentbased_traceidratio argument: "0.25" java: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest nodejs: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest python: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

Environment AWS EKS 1.27

Additional context apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: auto-instrumentation namespace: wingspan spec: exporter: endpoint: "http://observability-collector.aws-otel-eks.svc.cluster.local:4317/" propagators:

tracecontext
baggage
b3 sampler: type: parentbased_traceidratio argument: "0.25" java: image: repo/autoinstrumentation-java:1.0.0 nodejs: image: repo/autoinstrumentation-nodejs:1.0.0 python: image: repo//autoinstrumentation-python:1.0.0

after creating this getting

error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "pkg/scheme/scheme.go:28"

opentelemetry-operator: kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

mx-psi commented 9 months ago

cc @open-telemetry/operator-maintainers.

Could we transfer this to the operator repo? Maybe @jpkrohling can help here?

jaronoff97 commented 9 months ago

I do believe this is an operator issue. @shishir84 this log message implies you don't have the Instrumentation resource installed in your cluster. Have you made sure to install the operator CRDs?

shishir84 commented 9 months ago

I have installed the operator using https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml reference: https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file

shishir84 commented 9 months ago

applied annotation in java service: kubectl patch deploy/ab-deployment -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'
kubectl patch deploy/ab-deployment -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "true"}}}}}'

logs for instrumentation: kubectl logs otelinst/wn-instrumentation error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

otel logs: kubectl logs observability-collector-774957bc7d-5xxn9

2024-01-09T19:51:16.180Z info service@v0.91.0/telemetry.go:86 Setting up own telemetry... 2024-01-09T19:51:16.180Z info service@v0.91.0/telemetry.go:203 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} 2024-01-09T19:51:16.181Z info service@v0.91.0/service.go:145 Starting otelcol... {"Version": "0.91.0", "NumCPU": 8} 2024-01-09T19:51:16.181Z info extensions/extensions.go:34 Starting extensions... 2024-01-09T19:51:16.182Z warn internal@v0.91.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denia l of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/do cs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2024-01-09T19:51:16.182Z info otlpreceiver@v0.91.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0 .0.0:4317"} 2024-01-09T19:51:16.182Z warn internal@v0.91.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denia l of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/do cs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2024-01-09T19:51:16.182Z info otlpreceiver@v0.91.0/otlp.go:101 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoin t": "0.0.0.0:4318"} 2024-01-09T19:51:16.182Z info service@v0.91.0/service.go:171 Everything is ready. Begin running and processing data.

i am completely clueless being following the official documentation https://github.com/open-telemetry/opentelemetry-operator/tree/main?tab=readme-ov-file why such issues are coming.

jaronoff97 commented 9 months ago

that you are observing this log:

error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

implies that kubernetes does not have the instrumentation resource you are trying to apply. Can you try running kubectl get crd | grep 'instrumentation' and let me know the output?

shishir84 commented 9 months ago

kubectl get crd | grep 'instrumentation'

Output: instrumentations.opentelemetry.io 2024-01-09T18:57:24Z

shishir84 commented 9 months ago

Hi Team - can i get some solution. it is impacting our ongoing activities

jaronoff97 commented 9 months ago

Hello, I'm looking in to this but I am juggling many concurrent tickets. Please be patient with me as I investigate.

jaronoff97 commented 9 months ago

ah i see the problem now. You are attempting to get logs from the Instrumentation resource, however, the Instrumentation resource does not emit any logs at all. You most likely want logs from the application or the collector.

I confirmed this by installing an instrumentation resource locally, an otel sidecar, and an application to be instrumented w/ the sidecar. I then ran the following:

⫸ k logs otelinst/python
error: no kind "Instrumentation" is registered for version "opentelemetry.io/v1alpha1" in scheme "pkg/scheme/scheme.go:28"

I observed a working autoinstrumentation, however. Are you not seeing any traces out of your application? Can you confirm that the init container was successful?

shishir84 commented 9 months ago

its working now. thanks your your suggestion.

open-telemetry / opentelemetry-collector

auto instrumentation not working in eks 1.27 #9234