search

grafana / beyla

eBPF-based autoinstrumentation of web applications and network metrics
https://grafana.com/oss/beyla-ebpf/
Apache License 2.0
1.19k stars 77 forks source link

Fix rpc server address, should use otel.SpanHost instead of otel.Span.Peer #780

Closed corneliu-calciu closed 3 weeks ago

corneliu-calciu commented 3 weeks ago

The bug: rpc_system="grpc", server_address="backend-go", service_name="backend-go"

Actually server address is 10.152.183.18 port 50051. See below: [10.1.221.189 as backend-go]->[10.152.183.18 as 10.152.183.18:50051]

The server_address is wrong! Please see pkg/internal/export/prom/prom.go function: labelValuesHTTPClient is using otel.SpanHost(span) for the server address.

The logs: 2024-04-23 16:28:17.42342817 (148.711µs[101.119µs]) GRPC_SRV 0 /calculator.Calculator/SquareRoot [10.1.221.189 as backend-go]->[10.1.221.185 as ms-grpc-2:50051] size:0B svc=[default/ms-grpc-2 go] traceparent=[00-3d8c63d1f1cbbc4c3105477eaab79441-ee8936341d8f8576-01] 2024-04-23 16:28:17.42342817 (2.571143ms[2.571143ms]) GRPC_CLNT 0 /calculator.Calculator/SquareRoot [10.1.221.189 as backend-go]->[10.152.183.18 as 10.152.183.18:50051] size:0B svc=[default/backend-go go] traceparent=[00-3d8c63d1f1cbbc4c3105477eaab79441-374ff4797c48003f-01]

The exported metric: rpc_client_duration_seconds_count{instance="198.18.0.101:8999", job="beyladev", k8s_deployment_name="backend-go", k8s_namespace_name="apps", k8s_node_name="devenv", k8s_pod_name="backend-go-5d885f695c-cm7pq", k8s_pod_start_time="2024-04-23 16:20:23 +0000 UTC", k8s_pod_uid="53c19772-6339-4da5-93e3-85001f44f88f", k8s_replicaset_name="backend-go-5d885f695c", rpc_grpc_status_code="0", rpc_method="/calculator.Calculator/SquareRoot", rpc_system="grpc", server_address="backend-go", service_name="backend-go", service_namespace="default", target_instance="devenv-1331905"}

After the fix: rpc_client_duration_seconds_count{instance="198.18.0.101:8999", job="beyladev", k8s_deployment_name="backend-go", k8s_namespace_name="apps", k8s_node_name="devenv", k8s_pod_name="backend-go-5d885f695c-cm7pq", k8s_pod_start_time="2024-04-23 16:20:23 +0000 UTC", k8s_pod_uid="53c19772-6339-4da5-93e3-85001f44f88f", k8s_replicaset_name="backend-go-5d885f695c", rpc_grpc_status_code="0", rpc_method="/calculator.Calculator/SquareRoot", rpc_system="grpc", server_address="10.152.183.18", service_name="backend-go", service_namespace="default", target_instance="devenv-1331905"} rpc_system="grpc", server_address="10.152.183.18", service_name="backend-go"

Now the destination/server address is correct.

grcevski commented 3 weeks ago

Thanks for your contribution @corneliu-calciu!