In metrics for `envoy_cluster_upstream_rq` for label envoy_cluster_name , is showing wrong values `localhost:8080`

What happened?

Hi,

Please Note : With the previous version 1.8.0 this seems to be working fine, with no issue with the values.

Current Version details: Kuma version 2.2.0 Kubernetes version 1.26

The metrcis envoy_cluster_upstream_rq have a label envoy_cluster_namewhich is showing value as localhost:8080, for all pods running as a part of mesh.

I have been trying to refine a Grafana dashboard. Where I encountered that some of the metrics are not showing up correctly like envoy_cluster_upstream_rq , are showing the correct value for envoy_cluster_name but other metrics like envoy_cluster_upstream_rq_completed shows the correct the correct value. I have added both the metrics details.

envoy_cluster_upstream_rq (wrong)

{ 
 cluster_name="dev-cluster", 
 container="kuma-sidecar", 
 endpoint="metrics", 
 envoy_cluster_name="localhost_8080",  # wrong value
 envoy_response_code="200", 
 instance="100.72.234.64:5670", 
 job="prometheus-stack/kong-mesh-dp", 
 kuma_io_mesh_traffic="true" 
}

envoy_cluster_upstream_rq_completed (right)

{
 cluster_name="dev-cluster", 
 container="kuma-sidecar", 
 endpoint="metrics", 
 envoy_cluster_name="helloworld_green_svc_8080",  # right value
 instance="100.72.113.174:5670"
}

We have pod monitors to scrape metric.

Could you help if there is any specific config required.?

kumahq / kuma