Instrument crashed on Linux/arm64

[minimAluminiumalism]

Describe the bug

Failed to instrument on linux/arm64 with log:

{"level":"error","ts":1725711815.9766507,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:121","msg":"instrumentation crashed","error":"operation not permitted","stacktrace":"main.main\n\t/root/opentelemetry-go-instrumentation/cli/main.go:121\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}

Environment

linux/arm64 Ubuntu

To Reproduce

OTEL_GO_AUTO_TARGET_EXE=/root/server OTEL_SERVICE_NAME=eBPFApp OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=xxx ./otel-go-instrumentation

Expected behavior

Instrument successfully

Additional context

[MrAlias]

Did you run the CLI with elevated permissions (i.e. sudo)?

[MrAlias]

https://github.com/open-telemetry/opentelemetry-go-instrumentation/blob/dd08973247f7c3cfd453d54d49f335da48d8f5e6/docs/getting-started.md?plain=1#L38

[minimAluminiumalism]

Yes, I've tried it with sudo, and actually I ran the cmd with root in a docker image The base docker image is golang:1.22.6. Maybe you can reproduce the problem on this docker image?

[MrAlias]

Please provide a complete reproduction. The provided information does not specify what process is being targeted and how the auto-instrumentation is run (i.e. the provided command does not include sudo).

With what is provided, I am not able to reproduce the error. I am able to successfully run all of our examples locally and within containers.

[minimAluminiumalism]

I can run the demos in example and instrument them with docker compose up in getting-started.md, but I failed to instrument any golang app when I follow the instructions of Instrument an application on the same host of the file in a docker container.

I tested the auto-instrument program in a container bulit from golang:1.22.6 on Apple silicon Mac with M1 Pro.

1. Compile the otel-go-instrumentation binary: make build；
2. Write a simple HTTP web app and build it (the code is attached below) : GOOS=linux GOARCH=arm64 go build -o server main.go, so I get the executable binary server, and put it to directory /root;
3. Ran the app: ./server;
4. Try to instrument the server app: sudo OTEL_GO_AUTO_TARGET_EXE=/root/server OTEL_SERVICE_NAME=eBPFApp4 OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=[My_Endpoint_Here] ./otel-go-instrumentation;
5. Instrument failed with the log:

   {"level":"info","ts":1726057198.8453074,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:86","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
   {"level":"info","ts":1726057200.854215,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:67","msg":"found process","pid":294}
   {"level":"info","ts":1726057200.8722744,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:73","msg":"Detaching from process","pid":294}
   {"level":"info","ts":1726057200.8723752,"logger":"Instrumentation","caller":"opentelemetry-go-instrumentation/instrumentation.go:150","msg":"target process analysis completed","pid":294,"go_version":"1.22.2","dependencies":{"github.com/cespare/xxhash/v2":"2.1.2","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/gabriel-vasile/mimetype":"1.4.3","github.com/gin-contrib/sse":"0.1.0","github.com/gin-gonic/gin":"1.10.0","github.com/go-playground/locales":"0.14.1","github.com/go-playground/universal-translator":"0.18.1","github.com/go-playground/validator/v10":"10.20.0","github.com/go-redis/redis/v8":"8.11.5","github.com/leodido/go-urn":"1.4.0","github.com/mattn/go-isatty":"0.0.20","github.com/pelletier/go-toml/v2":"2.2.2","github.com/ugorji/go/codec":"1.2.12","golang.org/x/crypto":"0.23.0","golang.org/x/net":"0.25.0","golang.org/x/sys":"0.20.0","golang.org/x/text":"0.15.0","google.golang.org/protobuf":"1.34.1","gopkg.in/yaml.v3":"3.0.1","std":"1.22.2"},"total_functions_found":2}
   {"level":"info","ts":1726057200.8728652,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:119","msg":"starting instrumentation..."}
   {"level":"error","ts":1726057200.87302,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:121","msg":"instrumentation crashed","error":"operation not permitted","stacktrace":"main.main\n\t/root/opentelemetry-go-instrumentation/cli/main.go:121\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}

My app code:

package main

import (
    "context"
    "net/http"
    "os"

    "github.com/gin-gonic/gin"
    "github.com/go-redis/redis/v8"
)

var (
    rdb *redis.Client
    ctx = context.Background()
)

func jsonHandler(c *gin.Context) {
    c.JSON(http.StatusOK, gin.H{
        "message": "Hello, World!",
    })
}

func redisHandler(c *gin.Context) {
    key := "key"
    err := rdb.Set(ctx, key, "value", 0).Err()
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to set kv"})
        return
    }

    value, err := rdb.Get(ctx, key).Result()
    if err == redis.Nil {
        c.JSON(http.StatusNotFound, gin.H{"error": "Key not found"})
    } else if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    } else {
        c.JSON(http.StatusOK, gin.H{"key": key, "value": value})
    }
}

func main() {
    redisAddr := os.Getenv("REDIS_ADDR")
    if redisAddr == "" {
        redisAddr = "localhost:6379" // default value for local development
    }
    rdb = redis.NewClient(&redis.Options{
        Addr:     redisAddr,
        Password: "", // no password set
    })
    router := gin.Default()

    router.GET("/", jsonHandler)
    router.GET("/redis", redisHandler)
    router.Run(":8080")
}

[MrAlias]

If you are trying to run the eBPF auto-instrumentation agent directly on your Mac instead of in a container, it will fail given Macs do not have support for eBPF.

I'm happy to try reproducing your provided example if you can confirm you are running everything within a Linux container.

[minimAluminiumalism]

Yes, I've built the app binary and tried to instrument this app fully within a Debian container (ARM64) under macOS host.

[MrAlias]

Can you share your docker-compose or Dockerfile used?

[minimAluminiumalism]

Considering it's just for testing, I didn't use Docker Compose to deploy my app, and I executed commands directly inside the container with docker exec -it [container id] bash

[damemi]

Containers still rely on the host's kernel, so I don't think that is enough abstraction to get it to work natively on macos. Doing a quick search most articles recommend eBPF in a VM for mac, or additional steps to get it to work on docker desktop.

[minimAluminiumalism]

ok, I will try to instrument on a Linux VM.

[minimAluminiumalism]

Golang apps can be instrumented successfully in a Linux/ARM VM.