search

open-telemetry / opentelemetry-go-instrumentation

OpenTelemetry Auto Instrumentation using eBPF
https://opentelemetry.io
Apache License 2.0
485 stars 75 forks source link

Go-auto-instrumentation failling #619

Open fendrifirasleminnov opened 7 months ago

fendrifirasleminnov commented 7 months ago

Describe the bug

I am trying to add Go auto instrumentation to Rancher project , currently running in HA mode with 3 replicas.

Environment

To Reproduce

Steps to reproduce the behavior:

  1. Install OpenTelemetry Operator using this values.yaml 
    
manager:
image:
repository: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator
tag: v0.90.0
collectorImage:
repository: otel/opentelemetry-collector-contrib
tag: 0.90.1
autoInstrumentationImage:
go:
  repository: "otel/autoinstrumentation-go"
  tag: "v0.10.0-alpha"
featureGates: "+operator.autoinstrumentation.go"
certManager:
enabled: true

2. Install the collector

apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: rancher namespace: cattle-system spec: config: | receivers: otlp: protocols: grpc: http: processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 batch: send_batch_size: 10000 timeout: 10s

exporters:
  debug:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug]

3. Install Instrumentation Go

apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: rancher-instrumentation namespace: cattle-system spec: exporter: endpoint: http://rancher-collector-headless.cattle-system.svc.cluster.local:4317 propagators:

  1. Adding annotations
template:
    metadata:
      creationTimestamp: null
      labels:
        app: rancher
        release: rancher
      annotations:
        instrumentation.opentelemetry.io/inject-go: 'true'
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: /usr/bin/rancher
        sidecar.opentelemetry.io/inject: 'true'

Results

Instrumentation container is failing and restarting with a reason of "OOMKilled - exit code: 137" but in logs there is no error :

{"level":"info","ts":1705920290.1872249,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false} {"level":"info","ts":1705920292.1887016,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":53}

Also somehow this is affecting the main Rancher container that shows this logs: 2024/01/22 10:44:55 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.31]: websocket: bad handshake 2024/01/22 10:44:57 [INFO] Removing peer 10.42.0.33

Additional context

How can i verify if it's possible to add auto-instrumentation or this is a case where manual instrumentation with SDK is the only option ?

RonFed commented 7 months ago

Can you please attach the entire log?

fendrifirasleminnov commented 7 months ago

Sure @RonFed ,

This is the log of instrumentation sidecar and it just keeps restarting with same logs each time:

{"level":"info","ts":1705995691.3306305,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false}
{"level":"info","ts":1705995693.3312767,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":53}

This is the logs of Rancher container:

...

2024/01/23 07:41:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2024/01/23 07:41:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2024/01/23 07:41:28 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2024/01/23 07:41:28 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2024/01/23 07:41:28 [INFO] Starting cluster controllers for local
2024/01/23 07:41:28 [INFO] Starting /v1, Kind=Secret controller
2024/01/23 07:41:28 [INFO] Starting /v1, Kind=ServiceAccount controller
2024/01/23 07:41:28 [INFO] Starting cluster agent for local [owner=false]
2024/01/23 07:41:28 [INFO] Starting /v1, Kind=Namespace controller
2024/01/23 07:41:28 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2024/01/23 07:41:28 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2024/01/23 07:41:28 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2024/01/23 07:41:28 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2024/01/23 07:41:31 [INFO] Adding peer wss://10.42.0.31/v3/connect, 10.42.0.31
2024/01/23 07:41:31 [ERROR] Failed to connect to peer wss://10.42.0.31/v3/connect [local ID=10.42.0.32]: websocket: bad handshake
2024/01/23 07:41:34 [INFO] Removing peer 10.42.0.31

And that's it there is no direct error giving any clue about why instrumentation is failing.

zzz0615 commented 7 months ago

I had the same problem with OOMKilled.This happened to me too, with a pod OOMKilled shortly after the service started. But I've tested it across different services, and it doesn't happen in every one of them, and their environments are all the same

Environment Kubernetes 1.24 Go Version: 1.16 Version: v0.10.0-alpha

Here is my sidecar log: {"level":"info","ts":1706000131.0971994,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:94","msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false} {"level":"info","ts":1706000133.0977674,"logger":"Instrumentation.Analyzer","caller":"process/discover.go:69","msg":"found process","pid":7} {"level":"info","ts":1706000133.1378922,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.serverHandler.ServeHTTP","Offset":3245504,"ReturnOffsets":[3245676]}} {"level":"info","ts":1706000133.1380198,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"net/http.(Transport).roundTrip","Offset":3295200,"ReturnOffsets":[3295909,3296346,3296706,3296846,3297562,3297844,3297918,3298118,3298195,3298403,3298725,3298844,3299016,3299394,3299509]}} {"level":"info","ts":1706000133.1380327,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(DB).execDC","Offset":8297632,"ReturnOffsets":[8298137,8298489,8298913,8299132,8299275]}} {"level":"info","ts":1706000133.1380403,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"database/sql.(DB).queryDC","Offset":8300640,"ReturnOffsets":[8301476,8301677,8301795,8302392,8302538]}} {"level":"info","ts":1706000133.1380477,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(loopyWriter).headerHandler","Offset":14545440,"ReturnOffsets":[14545687,14545872,14545936,14546032,14546093,14546530]}} {"level":"info","ts":1706000133.138055,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(http2Client).NewStream","Offset":14598752,"ReturnOffsets":[14599205,14600416,14600542,14600701,14600816,14600888,14601764]}} {"level":"info","ts":1706000133.138061,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc/internal/transport.(http2Server).operateHeaders","Offset":14639456,"ReturnOffsets":[14640505,14641188,14641430,14641699,14643098,14644884,14647273,14651340,14651653,14651732]}} {"level":"info","ts":1706000133.138067,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(ClientConn).Invoke","Offset":14835296,"ReturnOffsets":[14835605,14835776]}} {"level":"info","ts":1706000133.1380723,"logger":"Instrumentation.Analyzer","caller":"process/analyze.go:98","msg":"found function","function_name":{"Name":"google.golang.org/grpc.(Server).handleStream","Offset":14931232,"ReturnOffsets":[14931931,14933377,14934190,14934484,14934587]}} {"level":"info","ts":1706000133.1382031,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":38} {"level":"info","ts":1706000133.138247,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":39} {"level":"info","ts":1706000133.1382716,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":40} {"level":"info","ts":1706000133.1382978,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":41} {"level":"info","ts":1706000133.138324,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":42} {"level":"info","ts":1706000133.1383505,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":43} {"level":"info","ts":1706000133.1383755,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":44} {"level":"info","ts":1706000133.1384165,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":45} {"level":"info","ts":1706000133.138443,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":46} {"level":"info","ts":1706000133.1384673,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":47} {"level":"info","ts":1706000133.13849,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":48} {"level":"info","ts":1706000133.1385124,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":49} {"level":"info","ts":1706000133.1385345,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":50} {"level":"info","ts":1706000133.1385741,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":51} {"level":"info","ts":1706000133.1386085,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":52} {"level":"info","ts":1706000133.138631,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":53} {"level":"info","ts":1706000133.1386545,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":54} {"level":"info","ts":1706000133.138681,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":55} {"level":"info","ts":1706000133.1387098,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":56} {"level":"info","ts":1706000133.138735,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":57} {"level":"info","ts":1706000133.1387618,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:133","msg":"attach successfully","tid":7} {"level":"info","ts":1706000133.1389458,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:234","msg":"Detected linux kernel version","version":"5.10.0"} {"level":"info","ts":1706000133.1390703,"logger":"Instrumentation.Allocate.TracedProgram","caller":"ptrace/ptrace_linux.go:246","msg":"mlock ret","ret":18446744073709551604} {"level":"info","ts":1706000133.1390805,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:64","msg":"Detaching from process","pid":7} {"level":"info","ts":1706000133.1391332,"logger":"Instrumentation.Allocate","caller":"process/allocate.go:43","msg":"mmaped remote memory","start_addr":"FFFFFFFFFFFFFFF5","end_addr":"7FFF5"} {"level":"info","ts":1706000133.1391432,"logger":"Instrumentation","caller":"app/instrumentation.go:134","msg":"target process analysis completed","pid":7,"go_version":"1.16.15","dependencies":{"boxspeed":null,"boxspeed/packages/orderserver/cmd":null,"cloud.google.com/go/compute":"1.6.1","cloud.google.com/go/monitoring":"0.1.0","cloud.google.com/go/trace":"0.1.0","contrib.go.opencensus.io/exporter/aws":"0.0.0-20200617204711-c478e41e60e9","contrib.go.opencensus.io/exporter/prometheus":"0.4.0","contrib.go.opencensus.io/exporter/stackdriver":"0.13.8","contrib.go.opencensus.io/exporter/zipkin":"0.1.2","contrib.go.opencensus.io/integrations/ocsql":"0.1.7","github.com/360EntSecGroup-Skylar/excelize":"1.4.1","github.com/KyleBanks/depth":"1.2.1","github.com/PuerkitoBio/purell":"1.1.1","github.com/PuerkitoBio/urlesc":"0.0.0-20170810143723-de5bf2ad4578","github.com/Shopify/sarama":"1.32.0","github.com/ThreeDotsLabs/watermill":"1.1.1","github.com/ThreeDotsLabs/watermill-kafka/v2":"2.2.2","github.com/XiaoMi/pegasus-go-client":"0.0.0-20210427083443-f3b6b08bc4c2","github.com/adhocore/gronx":"0.2.6","github.com/adjust/gorails":"0.0.0-20171013043634-2786ed0c03d3","github.com/akutz/memconn":"0.1.0","github.com/aliyun/alibaba-cloud-sdk-go":"1.61.1691","github.com/aliyun/aliyun-log-go-sdk":"0.1.36","github.com/aliyun/aliyun-oss-go-sdk":"2.2.7+incompatible","github.com/aliyun/aliyun-tablestore-go-sdk":"1.7.6","github.com/allegro/bigcache/v3":"3.0.2","github.com/andybalholm/brotli":"1.0.4","github.com/apache/thrift":"0.15.0","github.com/armon/go-metrics":"0.3.10","github.com/aws/aws-sdk-go":"1.40.34","github.com/beorn7/perks":"1.0.1","github.com/bmbstack/ripple":"1.2.3","github.com/bradfitz/gomemcache":"0.0.0-20220106215444-fb4bf637b56d","github.com/buger/jsonparser":"1.1.1","github.com/bytedance/sonic":"1.8.7","github.com/cenk/backoff":"2.2.1+incompatible","github.com/cenkalti/backoff":"2.2.1+incompatible","github.com/cenkalti/backoff/v3":"3.0.0","github.com/cenkalti/backoff/v4":"4.1.3","github.com/census-instrumentation/opencensus-proto":"0.3.0","github.com/cespare/xxhash/v2":"2.1.2","github.com/chenzhuoyu/base64x":"0.0.0-20221115062448-fe3a3abad311","github.com/chirino/graphql":"0.0.0-20210707003802-dfaf250c773e","github.com/clbanning/mxj":"1.8.4","github.com/cockroachdb/errors":"1.9.0","github.com/cockroachdb/logtags":"0.0.0-20211118104740-dabe8e521a4f","github.com/cockroachdb/redact":"1.1.3","github.com/cpuguy83/go-md2man/v2":"2.0.1","github.com/creasty/defaults":"1.6.0","github.com/davecgh/go-spew":"1.1.1","github.com/decred/dcrd/dcrec/secp256k1/v3":"3.0.0","github.com/dgrijalva/jwt-go":"3.2.0+incompatible","github.com/dgryski/go-jump":"0.0.0-20211018200510-ba001c3ffce0","github.com/dgryski/go-rendezvous":"0.0.0-20200823014737-9f7001d12a5f","github.com/dlclark/regexp2":"1.4.1-0.20201116162257-a2a8dda75c91","github.com/dop251/goja":"0.0.0-20210912140721-ac5354e9a820","github.com/dosco/graphjin":"0.17.0","github.com/dtm-labs/dtmcli":"1.14.2","github.com/dtm-labs/dtmdriver":"0.0.3","github.com/eapache/go-resiliency":"1.2.0","github.com/eapache/go-xerial-snappy":"0.0.0-20180814174437-776d5712da21","github.com/eapache/queue":"1.1.0","github.com/edwingeng/doublejump":"0.0.0-20210724020454-c82f1bcb3280","github.com/eko/gocache/v2":"2.3.1","github.com/facebookgo/clock":"0.0.0-20150410010913-600d898af40a","github.com/fatih/color":"1.13.0","github.com/flosch/pongo2/v4":"4.0.1","github.com/forgoer/openssl":"1.2.1","github.com/fsnotify/fsnotify":"1.5.4","github.com/garyburd/redigo":"1.6.3","github.com/getsentry/sentry-go":"0.12.0","github.com/go-co-op/gocron":"1.16.2","github.com/go-errors/errors":"1.0.1","github.com/go-http-utils/headers":"0.0.0-20181008091004-fed159eddc2a","github.com/go-kit/kit":"0.10.0","github.com/go-kit/log":"0.2.0","github.com/go-logfmt/logfmt":"0.5.1","github.com/go-logr/logr":"1.2.3","github.com/go-logr/stdr":"1.2.2","github.com/go-openapi/jsonpointer":"0.19.5","github.com/go-openapi/jsonreference":"0.19.6","github.com/go-openapi/spec":"0.20.4","github.com/go-openapi/swag":"0.19.15","github.com/go-pay/gopay":"1.5.78","github.com/go-ping/ping":"0.0.0-20211130115550-779d1e919534","github.com/go-pkgz/expirable-cache":"0.0.3","github.com/go-playground/locales":"0.14.0","github.com/go-playground/universal-translator":"0.18.0","github.com/go-playground/validator/v10":"10.11.0","github.com/go-redis/redis/v8":"8.11.5","github.com/go-resty/resty/v2":"2.7.0","github.com/go-sourcemap/sourcemap":"2.1.3+incompatible","github.com/go-sql-driver/mysql":"1.6.0","github.com/go-stack/stack":"1.8.0","github.com/gobuffalo/flect":"0.2.2","github.com/gogo/protobuf":"1.3.2","github.com/golang-jwt/jwt":"3.2.2+incompatible","github.com/golang/groupcache":"0.0.0-20210331224755-41bb18bfe9da","github.com/golang/mock":"1.6.0","github.com/golang/protobuf":"1.5.2","github.com/golang/snappy":"0.0.4","github.com/google/flatbuffers":"1.11.0","github.com/google/go-cmp":"0.5.8","github.com/google/go-querystring":"1.0.0","github.com/google/uuid":"1.3.0","github.com/googleapis/gax-go/v2":"2.4.0","github.com/gorilla/websocket":"1.4.2","github.com/grandcat/zeroconf":"1.0.0","github.com/guonaihong/gout":"0.3.7","github.com/hashicorp/consul/api":"1.12.0","github.com/hashicorp/errwrap":"1.1.0","github.com/hashicorp/go-cleanhttp":"0.5.2","github.com/hashicorp/go-hclog":"1.2.0","github.com/hashicorp/go-immutable-radix":"1.3.1","github.com/hashicorp/go-multierror":"1.1.1","github.com/hashicorp/go-rootcerts":"1.0.2","github.com/hashicorp/go-uuid":"1.0.2","github.com/hashicorp/golang-lru":"0.5.4","github.com/hashicorp/hcl":"1.0.0","github.com/hashicorp/serf":"0.9.7","github.com/imroc/req":"0.3.2","github.com/jackc/chunkreader/v2":"2.0.1","github.com/jackc/pgconn":"1.11.0","github.com/jackc/pgio":"1.0.0","github.com/jackc/pgpassfile":"1.0.0","github.com/jackc/pgproto3/v2":"2.2.0","github.com/jackc/pgservicefile":"0.0.0-20200714003250-2b9c44734f2b","github.com/jackc/pgtype":"1.10.0","github.com/jackc/pgx/v4":"4.15.0","github.com/jcmturner/aescts/v2":"2.0.0","github.com/jcmturner/dnsutils/v2":"2.0.0","github.com/jcmturner/gofork":"1.0.0","github.com/jcmturner/gokrb5/v8":"8.4.2","github.com/jcmturner/rpc/v2":"2.0.3","github.com/jinzhu/copier":"0.3.5","github.com/jinzhu/inflection":"1.0.0","github.com/jinzhu/now":"1.1.5","github.com/jmespath/go-jmespath":"0.4.0","github.com/josharian/intern":"1.0.0","github.com/json-iterator/go":"1.1.12","github.com/juju/ratelimit":"1.0.1","github.com/julienschmidt/httprouter":"1.3.0","github.com/jvatic/goja-babel":"0.0.0-20210918111805-77a268b37331","github.com/kardianos/osext":"0.0.0-20190222173326-2bc1f35cddc0","github.com/kavu/go_reuseport":"1.5.0","github.com/klauspost/compress":"1.15.1","github.com/klauspost/cpuid/v2":"2.0.9","github.com/kr/pretty":"0.3.0","github.com/kr/text":"0.2.0","github.com/labstack/echo/v4":"4.7.2","github.com/labstack/gommon":"0.3.1","github.com/leodido/go-urn":"1.2.1","github.com/lestrrat-go/backoff/v2":"2.0.7","github.com/lestrrat-go/httpcc":"1.0.0","github.com/lestrrat-go/iter":"1.0.0","github.com/lestrrat-go/jwx":"1.1.3","github.com/lestrrat-go/option":"1.0.0","github.com/lithammer/shortuuid/v3":"3.0.7","github.com/magiconair/properties":"1.8.6","github.com/mailru/easyjson":"0.7.7","github.com/mattn/go-colorable":"0.1.12","github.com/mattn/go-isatty":"0.0.14","github.com/matttproud/golang_protobuf_extensions":"1.0.2-0.20181231171920-c182affec369","github.com/miekg/dns":"1.1.45","github.com/mitchellh/hashstructure/v2":"2.0.2","github.com/mitchellh/mapstructure":"1.5.0","github.com/modern-go/concurrent":"0.0.0-20180306012644-bacd9c7ef1dd","github.com/modern-go/reflect2":"1.0.2","github.com/mohae/deepcopy":"0.0.0-20170929034955-c48cc78d4826","github.com/mozillazg/go-httpheader":"0.2.1","github.com/natefinch/lumberjack":"2.0.0+incompatible","github.com/oklog/ulid":"1.3.1","github.com/opentracing/opentracing-go":"1.2.0","github.com/openzipkin/zipkin-go":"0.2.4","github.com/pegasus-kv/thrift":"0.13.0","github.com/pelletier/go-toml/v2":"2.0.1","github.com/philhofer/fwd":"1.1.1","github.com/pierrec/lz4":"2.6.1+incompatible","github.com/pkg/errors":"0.9.1","github.com/prometheus/client_golang":"1.12.2","github.com/prometheus/client_model":"0.2.0","github.com/prometheus/common":"0.33.0","github.com/prometheus/procfs":"0.7.3","github.com/prometheus/statsd_exporter":"0.21.0","github.com/rcrowley/go-metrics":"0.0.0-20201227073835-cf1acfcdf475","github.com/robfig/cron/v3":"3.0.1","github.com/rogpeppe/go-internal":"1.8.1","github.com/rpcxio/libkv":"0.5.1-0.20210420120011-1fceaedca8a5","github.com/rs/cors":"1.8.2","github.com/rs/xid":"1.3.0","github.com/rubyist/circuitbreaker":"2.2.1+incompatible","github.com/russross/blackfriday/v2":"2.1.0","github.com/samuel/go-zookeeper":"0.0.0-20201211165307-7117e9ea2414","github.com/sensorsdata/sa-sdk-go":"2.1.0+incompatible","github.com/shopspring/decimal":"1.2.0","github.com/sirupsen/logrus":"1.8.1","github.com/smallnest/rpcx":"1.7.3","github.com/soheilhy/cmux":"0.1.5","github.com/spf13/afero":"1.8.2","github.com/spf13/cast":"1.5.0","github.com/spf13/jwalterweatherman":"1.1.0","github.com/spf13/pflag":"1.0.5","github.com/spf13/viper":"1.12.0","github.com/subosito/gotenv":"1.3.0","github.com/swaggo/echo-swagger":"1.3.2","github.com/swaggo/files":"0.0.0-20210815190702-a29dd2bc99b2","github.com/swaggo/swag":"1.8.1","github.com/tencentcloud/tencentcloud-cls-sdk-go":"1.0.2","github.com/tencentyun/cos-go-sdk-v5":"0.7.38","github.com/thoas/go-funk":"0.9.2","github.com/tidwall/gjson":"1.14.1","github.com/tidwall/match":"1.1.1","github.com/tidwall/pretty":"1.2.0","github.com/tinylib/msgp":"1.1.6","github.com/twitchyliquid64/golang-asm":"0.15.1","github.com/ulule/limiter/v3":"3.10.0","github.com/urfave/cli/v2":"2.6.0","github.com/valyala/bytebufferpool":"1.0.0","github.com/valyala/fastrand":"1.1.0","github.com/valyala/fasttemplate":"1.2.1","github.com/vmihailenco/msgpack":"4.0.4+incompatible","github.com/vmihailenco/msgpack/v5":"5.3.5","github.com/vmihailenco/tagparser/v2":"2.0.0","github.com/xdg-go/pbkdf2":"1.0.0","github.com/xdg-go/scram":"1.1.0","github.com/xdg-go/stringprep":"1.0.2","github.com/youmark/pkcs8":"0.0.0-20181117223130-1be2e3e5546d","go.mongodb.org/mongo-driver":"1.8.3","go.opencensus.io":"0.23.0","go.opentelemetry.io/contrib/instrumentation/github.com/Shopify/sarama/otelsarama":"0.31.0","go.opentelemetry.io/otel":"1.6.1","go.opentelemetry.io/otel/trace":"1.6.1","go.uber.org/atomic":"1.9.0","go.uber.org/multierr":"1.7.0","go.uber.org/zap":"1.19.1","golang.org/x/arch":"0.0.0-20210923205945-b76863e36670","golang.org/x/crypto":"0.0.0-20220411220226-7b82a4e95df4","golang.org/x/net":"0.0.0-20220520000938-2e3eb7b945c2","golang.org/x/oauth2":"0.0.0-20220411215720-9780585627b5","golang.org/x/sync":"0.0.0-20220513210516-0976fa681c29","golang.org/x/sys":"0.0.0-20220520151302-bc2c85ada10a","golang.org/x/text":"0.3.7","golang.org/x/time":"0.0.0-20210723032227-1f47c861a9ac","golang.org/x/tools":"0.1.10","google.golang.org/api":"0.81.0","google.golang.org/genproto":"0.0.0-20220519153652-3a47de7e79bd","google.golang.org/grpc":"1.46.2","google.golang.org/protobuf":"1.28.0","gopkg.in/guregu/null.v4":"4.0.0","gopkg.in/ini.v1":"1.66.4","gopkg.in/natefinch/lumberjack.v2":"2.0.0","gopkg.in/resty.v1":"1.12.0","gopkg.in/tomb.v2":"2.0.0-20161208151619-d5d1b5820637","gopkg.in/yaml.v2":"2.4.0","gopkg.in/yaml.v3":"3.0.1","gorm.io/driver/mysql":"1.3.3","gorm.io/driver/postgres":"1.3.4","gorm.io/gorm":"1.23.4","k8s.io/apimachinery":"0.23.5","std":"1.16.15"},"total_functions_found":9} {"level":"info","ts":1706000133.1395338,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:112","msg":"no functions found for probe, removing","name":"github.com/gin-gonic/gin/server"} {"level":"info","ts":1706000133.1395707,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."} {"level":"info","ts":1706000133.1397038,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:186","msg":"Mounting bpffs","allocations_details":{"StartAddr":18446744073709551605,"EndAddr":524277}} {"level":"info","ts":1706000133.1413236,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:172","msg":"loading probe","name":"google.golang.org/grpc/client"} {"level":"info","ts":1706000133.2181513,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:172","msg":"loading probe","name":"google.golang.org/grpc/server"} {"level":"info","ts":1706000133.2567494,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:172","msg":"loading probe","name":"net/http/server"} {"level":"info","ts":1706000133.5480084,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:172","msg":"loading probe","name":"net/http/client"} {"level":"info","ts":1706000133.5801952,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:172","msg":"loading probe","name":"database/sql/client"} {"level":"info","ts":1706000133.586491,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:180","msg":"loaded probes to memory","total_probes":5} {"level":"info","ts":1706000135.9823651,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000139.7868557,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/riskService/v1.1"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000139.787365,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":40816}}]} {"level":"info","ts":1706000139.7923002,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":40818}}]} {"level":"info","ts":1706000140.9866405,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000141.8746693,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/logstores/stage-boxlite-orderserver-service"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000145.9910016,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000149.801416,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/riskService/v1.1"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000149.8019218,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":43148}}]} {"level":"info","ts":1706000150.9954832,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000151.923856,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/logstores/stage-boxlite-orderserver-service"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000154.7958796,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/riskService/v1.1"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000154.796441,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":60738}}]} {"level":"info","ts":1706000155.9996345,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000156.8389945,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/logstores/stage-boxlite-orderserver-service"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000159.8015192,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/riskService/v1.1"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000159.802031,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":60740}}]} {"level":"info","ts":1706000161.003882,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000161.8686302,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/logstores/stage-boxlite-orderserver-service"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000166.0078685,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"net.peer.port","Value":{"Type":"INT64","Value":9848}},{"Key":"rpc.system","Value":{"Type":"STRING","Value":"grpc"}},{"Key":"rpc.service","Value":{"Type":"STRING","Value":"/Request/request"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"mse-63c9f4e2-nacos-ans.mse.aliyuncs.com"}}]} {"level":"info","ts":1706000169.785675,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/riskService/v1.1"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":0}}]} {"level":"info","ts":1706000169.7861686,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":35554}}]} {"level":"info","ts":1706000169.7941275,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/ping"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"net.host.name","Value":{"Type":"STRING","Value":"192.168.130.125:31003"}},{"Key":"net.protocol.name","Value":{"Type":"STRING","Value":"HTTP/1.1"}},{"Key":"net.peer.name","Value":{"Type":"STRING","Value":"192.168.30.8"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":35560}}]}

iblancasa commented 2 months ago

Same situation here.

I noticed this happens when the executable is big.