No status code for a go app

[dirtyren]

Describe the bug I have installed Odigos v1.0.1 o a GKE Cluster and instrumented a Go app with it and although I can see the tracings, none of them have the status code on signoz. The app handles internally all the status code in order for us to check our own SLOs, so I find it strange signoz is no showing them.

Screenshots This a the go app sending data to victoria metricas

This is the same app talking to Pub/Sub

Just to add another Go app instrumented by signoz that is showing the status code.

Desktop (please complete the following information):

• Google GKE

[RonFed]

Are the missing status codes occurring for HTTP server or client application?

[dirtyren]

Hey @RonFed , client application. In this case is a Go app querying data from via victoria metrics through a Google LB. Our API behind a nginx-ingress is returning status code 200 as it should and Odigos is showing it right. Tks.

[RonFed]

@dirtyren Thanks. Once https://github.com/open-telemetry/opentelemetry-go-instrumentation/pull/527 is merged, we'll update Odigos and this should be fixed

[RonFed]

@dirtyren with the latest v1.0.10 release, this should be resolved.

[dirtyren]

@RonFed I noticed that open-telemetry merged the code and I just updated odigos to the latest version 1.0.10 and I am still not getting the status code for the tracings. Just to let you know. Tks.

[RonFed]

@dirtyren Which Go version do you use ? There suppose to be another release to the OpenTelemetry go instrumentation supporting the newest Go versions (v1.20.12 v1.21.4 v1.21.5)

[dirtyren]

@RonFed we use Go 1.20 on the deploy I am instrumenting with Odigos.

Just adding a screenshot to show you how I am seeing the tracings on Signoz

[dirtyren]

@RonFed I also noticed this happening after upgrading to 1.0.10. New instrumented applications using the odigos-ui do not have the same annotations that previous instrumented apps and, because of that, are not catching tracings.

Anything you need just let me know.

kubectl get deploy komodo-api -o yaml | grep odigo
    instrumentation.odigos.io/ebpf: "true"
    odigos-instrumentation: enabled
            instrumentation.odigos.io/go: "1"

kubectl get deploy komodo-frontend -o yaml | grep odigo
    odigos-instrumentation: enabled

[blumamir]

@RonFed I also noticed this happening after upgrading to 1.0.10. New instrumented applications using the odigos-ui do not have the same annotations that previous instrumented apps and, because of that, are not catching tracings.

Anything you need just let me know.

kubectl get deploy komodo-api -o yaml | grep odigo
    instrumentation.odigos.io/ebpf: "true"
    odigos-instrumentation: enabled
            instrumentation.odigos.io/go: "1"

kubectl get deploy komodo-frontend -o yaml | grep odigo
    odigos-instrumentation: enabled

Hi @dirtyren ,

Thanks for letting us know.

Can you please check to see if you have an instrumented application for this source?

kubectl get instrumentedapplications.odigos.io -A

and also please check the logs of theodigos-instrumentor pod in the odigos-system namespace to see if there is any clue about the issue?

[dirtyren]

Hello @RonFed ,

the app komodo-frontend is a react frontend, that I am not able to instrument using Odigos

kubectl get instrumentedapplications.odigos.io -A
NAMESPACE   NAME                       AGE
komodo      deployment-komodo-api      28d
komodo      deployment-slo-generator   28d

I could not find any messages on the odigos-instrumentor pod. Also, I had to remove the label odigos-instrumentation: enabled from the komodo-frontend deploy so it would show on the web interface and I could redo the process to check for logs.

Anything else I can help you with?

Tks.

[blumamir]

Hello @RonFed ,

the app komodo-frontend is a react frontend, that I am not able to instrument using Odigos

kubectl get instrumentedapplications.odigos.io -A
NAMESPACE   NAME                       AGE
komodo      deployment-komodo-api      28d
komodo      deployment-slo-generator   28d

I could not find any messages on the odigos-instrumentor pod. Also, I had to remove the label odigos-instrumentation: enabled from the komodo-frontend deploy so it would show on the web interface and I could redo the process to check for logs.

Anything else I can help you with?

Tks.

Thanks for the info.

I think I found the bug. I am working on a fix that should be ready soon 🙏

[blumamir]

@dirtyren

I thought I found an issue but I am not sure. Can you please try to upgrade to version v1.0.11 and check if you still encounter the issue?

I also wrote to you in odigos slack - if you are available we can jump on a quick call to help diagnose the issue on your setup. Please let me know if the new version fixes the issue and/or if you like to schedule a zoom call.

Thanks again 🙏

[dirtyren]

hey @blumamir , I just updated to v1.0.11 and the service names of my instrumented apps got weird like this:

The tracing information is also showing strange things, but, it seems to be showing the http codes

The service names were working before the update to 1.0.11.

Anything I can help with?

Tks.

[dirtyren]

I am getting this errors on the odiglet logs

{"level":"error","ts":1703707677.1487029,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:165","msg":"error while loading probes, cleaning up","name":"google.golang.org/grpc/server","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: permission denied: dereference of modified ctx ptr R1 off=40 disallowed (33 line(s) omitted)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/root/go/pkg/mod/go.opentelemetry.io/auto@v0.9.0-alpha/internal/pkg/instrumentation/manager.go:165\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/root/go/pkg/mod/go.opentelemetry.io/auto@v0.9.0-alpha/internal/pkg/instrumentation/manager.go:120\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/root/go/pkg/mod/go.opentelemetry.io/auto@v0.9.0-alpha/instrumentation.go:152\ngithub.com/keyval-dev/odigos/odiglet/pkg/ebpf.(*GoOtelEbpfSdk).Run\n\t/go/src/github.com/keyval-dev/odigos/odiglet/pkg/ebpf/go.go:55\ngithub.com/keyval-dev/odigos/odiglet/pkg/ebpf.(*EbpfDirector[...]).Instrument.func1\n\t/go/src/github.com/keyval-dev/odigos/odiglet/pkg/ebpf/director.go:124"}
{"level":"info","ts":1703707677.2024436,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:190","msg":"Cleaning bpffs"}
{"level":"error","ts":1703707677.2028763,"caller":"ebpf/director.go:125","msg":"instrumentation crashed after running","error":"field UprobeHttp2ServerOperateHeader: program uprobe_http2Server_operateHeader: load program: permission denied: dereference of modified ctx ptr R1 off=40 disallowed (33 line(s) omitted)","stacktrace":"github.com/keyval-dev/odigos/odiglet/pkg/ebpf.(*EbpfDirector[...]).Instrument.func1\n\t/go/src/github.com/keyval-dev/odigos/odiglet/pkg/ebpf/director.go:125"}
{"level":"info","ts":1703762962.9726727,"caller":"ebpf/director.go:145","msg":"Cleaning up ebpf go instrumentation for pod","pod":{"name":"komodo-api-658fdb84b-6zhjj","namespace":"komodo"}}
a

[edeNFed]

Hi @dirtyren do you still need help with this issue? Happy to help you debug this if needed

[blumamir]

@dirtyren if the issue is still relevant, feel free to re-open