search

grafana / beyla

eBPF-based autoinstrumentation of web applications and network metrics
https://grafana.com/oss/beyla-ebpf/
Apache License 2.0
1.19k stars 77 forks source link

panic: label value "***" is not valid UTF-8 still exist in v1.4.1 #746

Closed icylord closed 4 weeks ago

icylord commented 4 weeks ago

version: 1.4.1, panic still exist, seems the fix ref from https://github.com/grafana/beyla/issues/618 did not completely be solved?

event type 3, protocol 2
time=2024-04-17T01:59:20.514Z level=DEBUG msg="submitting traces on timeout" component=ringbuf.Tracer len=60
2024-04-17 01:59:19.41715919 (2.503734ms[2.503734ms]) 200 GET /apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-snapshotter-leader-driver-lo [
]->[10.43.0.1:443] size:0B svc=[longhorn-system/csi-snapshotter go] traceparent=[00-530238ba7759c81e644a18f0aedef865-0000000000000000-01]
2024-04-17 01:59:19.41715919 (2.806791ms[2.806791ms]) 200 GET /apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-attacher-leader-driver-longh []->[10.43.0.1:443] size:0B svc=[longhorn-system/csi-attacher go] traceparent=[00-30301a1d6fe8bc9c8d300a2928195fbe-0000000000000000-01]
2024-04-17 01:59:19.41715919 (3.642235ms[3.642235ms]) 200 PUT /apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-attacher-leader-driver-longh [²MIOj†¯¾û��¤«T]->[10.43.0.1:443] size:750B svc=[longhorn-system/csi-attacher go] traceparent=[00-848671f2e2e829bf3795b88108649195-0000000000000000-01]
2024-04-17 01:59:20.41715920 (126.335µs[88.513µs]) 0 ŴNb
 /grpc.health.v1.Health/Check [10.42.71.97]->[10.42.71.100:8501] size:0B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-acd5ef838d4631c1a281bbb937a4b531-0000000000000000-01]
2024-04-17 01:59:20.41715920 (292.426µs[292.426µs]) 0 ˆšʈ²4ڏ0@‰šʈµ%B1 ƒ†E•bk+"󔴦uû…|e   ‹ӔAŒhKº \y°_‹  [10.42.71.97]->[10.42.71.100:8501] size:135B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[]
2024-04-17 01:59:20.41715920 (1.450197ms[1.450197ms]) 0 p,􎢊 /grpc.health.v1.Health/Check []->[10.42.71.100:8501] size:3564088310662853432B svc=[longhorn-system/longhorn-manager go] traceparent=[00-9b68e9507bf26c1a38aff7cfe68151f5-0000000000000000-01]
2024-04-17 01:59:20.41715920 (137.588µs[108.416µs]) 0 ¹#M¨A /ptypes.ControllerService/ReplicaList [10.42.71.100]->[10.42.71.100:10060] size:0B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-9aeb35f88ad4357bc0129be5564ef6a0-0000000000000000-01]
2024-04-17 01:59:20.41715920 (349.456µs[349.456µs]) 0   [10.42.71.100]->[10.42.71.100:10060] size:207B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[]
2024-04-17 01:59:20.41715920 (1.185701ms[1.185701ms]) 0 ü􎢊 /ptypes.ControllerService/ReplicaList []->[10.42.71.100:10060] size:523021969416428169B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-defd5e128e0fd7082c63ba7c12de2ef8-a016dc3ab53066a0-01]
2024-04-17 01:59:20.41715920 (1.709902ms[1.696624ms]) 0  /imrpc.ProxyEngineService/ReplicaList [10.42.71.97]->[10.42.71.100:8501] size:1081991571664033802B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-defd5e128e0fd7082c63ba7c12de2ef8-0000000000000000-01]
2024-04-17 01:59:20.41715920 (1.820156ms[1.820156ms]) 0 POST /imrpc.ProxyEngineService/ReplicaList [10.42.71.97]->[10.42.71.100:8501] size:168B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[]
2024-04-17 01:59:20.41715920 (2.093611ms[2.093611ms]) 0  /imrpc.ProxyEngineService/ReplicaList []->[10.42.71.100:8501] size:8243086452505012033B svc=[longhorn-system/longhorn-manager go] traceparent=[00-364df6b281764751ca07b4deb9c2323a-0000000000000000-01]
2024-04-17 01:59:20.41715920 (197.988µs[153.308µs]) 0 ȡ‹Nb
 /ptypes.ControllerService/ReplicaList [10.42.71.100]->[10.42.71.100:10060] size:7301290945786900037B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-ba1625fe412d4b00a0083583b6ad5cff-0000000000000000-01]
2024-04-17 01:59:20.41715920 (476.768µs[476.768µs]) 0 POST /ptypes.ControllerService/ReplicaList [10.42.71.100]->[10.42.71.100:10060] size:216B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[]
2024-04-17 01:59:20.41715920 (1.106457ms[1.106457ms]) 0  /ptypes.ControllerService/ReplicaList []->[10.42.71.100:10060] size:7301290945786900037B svc=[longhorn-system/instance-manager-c3edaea05c2f0bf1d56758185a15f624 go] traceparent=[00-9321989de3d6d5733750abb499b76999-dd7c64431b709aec-01]
panic: label value "\x88\x9a\xcaȲ\x124ڏ\x010@\x89\x9a\xcaȵ%B\a1\x7f\x00\x83\x86E\x95bk+\"\xf3\x94t&u\xfb\x85|e\x1d\t\x9d\x8bӔ\x9d\x7fA\x8c\b\x17hK\xba\x15\xc2\x00\\y\xb0\a_\x8b" is not valid UTF-8

goroutine 130 [running]:
github.com/prometheus/client_golang/prometheus.(*HistogramVec).WithLabelValues(...)
    /opt/app-root/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:1200
github.com/grafana/beyla/pkg/internal/export/prom.(*metricsReporter).observe(0xc0002f2070, 0xc003a54c60)
    /opt/app-root/pkg/internal/export/prom/prom.go:231 +0x996
github.com/grafana/beyla/pkg/internal/export/prom.(*metricsReporter).reportMetrics(0xc0002f2070, 0x0?)
    /opt/app-root/pkg/internal/export/prom/prom.go:219 +0x110
github.com/mariomac/pipes/pkg/node.(*Terminal[...]).start.func1()
    /opt/app-root/vendor/github.com/mariomac/pipes/pkg/node/node.go:193 +0x2d
created by github.com/mariomac/pipes/pkg/node.(*Terminal[...]).start in goroutine 41
    /opt/app-root/vendor/github.com/mariomac/pipes/pkg/node/node.go:192 +0x6a
grcevski commented 4 weeks ago

Thanks for reporting it, I will take a look.

grcevski commented 4 weeks ago

I think this is an issue with our HTTP2 support. The related problem is this one https://github.com/grafana/beyla/issues/742. We only support golang.org/x/net to version 0.21 and there were some breaking changes in the API. I see that longhorn mentioned here is on version 0.22.

I'll prioritize fixing this.

icylord commented 4 weeks ago

I think this is an issue with our HTTP2 support. The related problem is this one #742. We only support golang.org/x/net to version 0.21 and there were some breaking changes in the API. I see that longhorn mentioned here is on version 0.22.

I'll prioritize fixing this.

thanking for replying, the longhorn version is 1.5.4, and the version of golang.org/x/net is v0.20.0 in go.mod in that branch, ref https://github.com/longhorn/longhorn-manager/blob/v1.5.4/go.mod https://github.com/longhorn/longhorn-instance-manager/blob/v1.5.4/go.mod https://github.com/longhorn/longhorn-engine/blob/v1.5.4/go.mod

grcevski commented 4 weeks ago

Thanks for the info! I did find one more issue and that's also in the PR. Once it's merged in main, I'll backport it and we'll make a new 1.4 release.