search

odigos-io / opentelemetry-go-instrumentation

OpenTelemetry auto-instrumentation for Go applications
Apache License 2.0
292 stars 44 forks source link

Error during instrumentation: `invalid argument: ;` #35

Open ov7a opened 2 years ago

ov7a commented 2 years ago

Expected Behavior

Auto Instrumentation works without issues.

Actual Behavior

Sidecar container halts with error.

{"level":"info","ts":1666009431.26208,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1666009431.2621613,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1666009433.2638783,"caller":"process/discover.go:42","msg":"found process","pid":13}
{"level":"info","ts":1666009433.2643228,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":139872935333888}
{"level":"info","ts":1666009433.281399,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1666009433.2829392,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}
{"level":"info","ts":1666009433.2831843,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2}
{"level":"info","ts":1666009433.2836869,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5}
{"level":"info","ts":1666009433.2840135,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.17.11","dependencies":{"github.com/FZambia/eagle":"v0.0.2","github.com/FZambia/sentinel":"v1.1.0","github.com/FZambia/statik":"v0.1.2-0.20180217151304-b9f012bb2a1b","github.com/FZambia/tarantool":"v0.3.1","github.com/FZambia/viper-lite":"v0.0.0-20220110144934-1899f66c7d0e","github.com/beorn7/perks":"v1.0.1","github.com/centrifugal/centrifuge":"v0.23.1","github.com/centrifugal/protocol":"v0.8.8","github.com/cespare/xxhash/v2":"v2.1.2","github.com/cristalhq/jwt/v4":"v4.0.0","github.com/gobwas/glob":"v0.2.3","github.com/golang/protobuf":"v1.5.2","github.com/gomodule/redigo":"v1.8.8","github.com/google/uuid":"v1.3.0","github.com/gorilla/securecookie":"v1.1.1","github.com/gorilla/websocket":"v1.5.0","github.com/igm/sockjs-go/v3":"v3.0.2","github.com/josharian/intern":"v1.0.0","github.com/mailru/easyjson":"v0.7.7","github.com/mattn/go-isatty":"v0.0.12","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/mitchellh/mapstructure":"v1.4.3","github.com/mna/redisc":"v1.3.2","github.com/nats-io/nats.go":"v1.14.0","github.com/nats-io/nkeys":"v0.3.0","github.com/nats-io/nuid":"v1.0.1","github.com/pelletier/go-toml":"v1.9.4","github.com/prometheus/client_golang":"v1.12.1","github.com/prometheus/client_model":"v0.2.0","github.com/prometheus/common":"v0.34.0","github.com/prometheus/procfs":"v0.7.3","github.com/rakutentech/jwk-go":"v1.0.1","github.com/rs/zerolog":"v1.21.0","github.com/segmentio/asm":"v1.1.4","github.com/segmentio/encoding":"v0.3.5","github.com/spf13/cast":"v1.4.1","github.com/spf13/cobra":"v0.0.7","github.com/spf13/jwalterweatherman":"v1.1.0","github.com/spf13/pflag":"v1.0.5","github.com/valyala/bytebufferpool":"v1.0.0","github.com/vmihailenco/msgpack/v5":"v5.3.5","github.com/vmihailenco/tagparser/v2":"v2.0.0","golang.org/x/crypto":"v0.0.0-20220411220226-7b82a4e95df4","golang.org/x/net":"v0.0.0-20220421235706-1d1ef9303861","golang.org/x/sync":"v0.0.0-20210220032951-036812b2e83c","golang.org/x/sys":"v0.0.0-20220422013727-9388b58f7150","golang.org/x/text":"v0.3.7","google.golang.org/genproto":"v0.0.0-20220422154200-b37d22cd5731","google.golang.org/grpc":"v1.46.0","google.golang.org/protobuf":"v1.28.0","gopkg.in/yaml.v2":"v2.4.0"},"total_functions_found":4}
{"level":"info","ts":1666009433.2841408,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1666009433.312439,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":139872935333888,"end_addr":139872947916800}
{"level":"info","ts":1666009433.3126094,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"google.golang.org/grpc"}
{"level":"info","ts":1666009433.3144336,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":139872947916800,"is_registers_abi":true,"start_addr":139872935333888,"total_cpus":4}}
{"level":"error","ts":1666009433.315479,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).load\n\t/app/pkg/instrumentors/runner.go:71\ngithub.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).Run\n\t/app/pkg/instrumentors/runner.go:19\nmain.main\n\t/app/cli/main.go:71\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1666009433.3155577,"caller":"grpc/probe.go:213","msg":"closing gRPC instrumentor"}
{"level":"info","ts":1666009433.3155837,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"error","ts":1666009433.315593,"caller":"cli/main.go:73","msg":"error while running instrumentors","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"main.main\n\t/app/cli/main.go:73\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Steps to Reproduce the Problem

I've followed the tutorial and tried to add instrumentation to an existing deployment in k8s. Here's a part of deployment.yaml, it is basically a copy of diff between emojivoto and emojivoto-instrumented

...
    spec:
      shareProcessNamespace: true
      initContainers:
        - name: copy-launcher
          image: keyval/launcher:v0.1
          command:
            - cp
            - -a
            - /kv-launcher/.
            - /odigos-launcher/
          volumeMounts:
            - name: launcherdir
              mountPath: /odigos-launcher
      volumes:
        - name: launcherdir
          emptyDir: { }
        - name: kernel-debug
          hostPath:
            path: /sys/kernel/debug
      containers:
        - name: ...
          command:
            - /odigos-launcher/launch
            - /usr/local/bin/centrifugo
            - --health
            - --prometheus
          ...
        - name: instrumentation
          image: keyval/otel-go-agent:v0.6.0
          env:
            - name: OTEL_TARGET_EXE
              value: /usr/local/bin/centrifugo
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "otel-collector:4317"
            - name: OTEL_SERVICE_NAME
              value: "test"
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true
          volumeMounts:
            - mountPath: /sys/kernel/debug
              name: kernel-debug

Additional Info