open-telemetry / opentelemetry-go-instrumentation

OpenTelemetry Auto Instrumentation using eBPF
https://opentelemetry.io
Apache License 2.0
532 stars 81 forks source link

OMMKilled container after being instrumented #902

Open iblancasa opened 4 months ago

iblancasa commented 4 months ago

Describe the bug

I'm trying to autoinstrument the OpenTelemetry Collector. I tried copying the binary from the otel/opentelemetry-collector image to another based on fedora.

from otel/opentelemetry-collector as origin
from fedora
COPY --from=origin /otelcol /otelcol
ENTRYPOINT ["/otelcol"]

When I try to instrument this container image, the container is OOMKilled:

{"level":"info","ts":1719841365.371024,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1719841367.371693,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:78","msg":"found process","pid":7}
{"level":"info","ts":1719841367.3726802,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:148","msg":"No symbols found in binary, trying to find functions using .gosymtab"}

When I build my own distribution with only the otlp receiver and debug exporter, the instrumentation progresses a little bit more but it is also OOMKilled. This is the log I get:

{"level":"info","ts":1719839515.3163264,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:140","msg":"no functions found for probe, removing","name":"github.com/segmentio/kafka-go/consumer"}
{"level":"info","ts":1719839515.316356,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:140","msg":"no functions found for probe, removing","name":"database/sql/client"}
{"level":"info","ts":1719839515.316361,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:140","msg":"no functions found for probe, removing","name":"github.com/segmentio/kafka-go/producer"}
{"level":"info","ts":1719839515.3163679,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:117","msg":"starting instrumentation..."}
{"level":"info","ts":1719839515.3164241,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:222","msg":"Mounting bpffs","allocations_details":{"StartAddr":140352138248192,"EndAddr":140352138772480,"NumCPU":16}}
{"level":"info","ts":1719839515.3165295,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:208","msg":"loading probe","name":"google.golang.org/grpc/client"}

Expected behavior

The container should be instrumented without any error.

dartagnanli commented 5 days ago

I got this too,is there any solution about this?

iblancasa commented 5 days ago

@dartagnanli I was able to make it work (more or less) reducing the number of objects in the binary.

damemi commented 5 days ago

@iblancasa that's interesting, could you share more of what you mean by that and how you did it?

iblancasa commented 5 days ago

As you can see, I was trying to inject the instrumentation in the opentelemetry-collector. I just removed all the receiver/exporters except the otlp receiver.

Long time without working on this but I think all the probes were being loaded in memory and that was a problem..