Open diranged opened 3 months ago
Pinging code owners:
pkg/resourcetotelemetry: @mx-psi
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Attaching the relevent memorylimiter logs... Explore-logs-2024-06-04 13_14_33.txt
Could there be some metrics that have a very large number of resource attributes? We end up allocating extra memory of size roughly $\textrm{number of metrics} \times \textrm{avg number of resource attributes}$, so if the number of metrics is not too big, maybe the number of resource attributes explains this.
I am a bit skeptical of this being an issue on pkg/resourcetotelemetry
at first, there may be room for improvement but the logic there is pretty simple https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/e7cf56040c1bf33599f149594d9beea6dcc4cb8e/pkg/resourcetotelemetry/resource_to_telemetry.go#L108-L112
and it looks like it allocates exactly what it needs.
@mx-psi,
Thanks for the quick response. We do add a decent number of resource attributes to every metric. At this time, we're only dealing with processing metrics collected by the prometheusreceiver
. These metrics each get a bunch of standard labels applied by the k8sattributesprocessor
:
app_env="xxx",
app_group="xxx",
app_kubernetes_io_instance="xxx",
app_kubernetes_io_managed_by="Helm",
app_kubernetes_io_name="xx",
cloud_account_id="xxx",
cloud_availability_zone="xxx",
cloud_platform="aws_eks",
cloud_provider="aws",
cloud_region="eu-west-1",
cluster="eu1",
component="proxy",
container="istio-proxy",
container_id="xxx",
container_image_name="xxx/istio/proxyv2",
container_image_tag="1.20.6",
host_arch="amd64",
host_id="i-xxx",
instance="xxx:15090",
job="istio-system/envoy-stats-monitor-raw",
k8s_cluster_name="eu1",
k8s_container_name="istio-proxy",
k8s_deployment_name="xxx",
k8s_namespace_name="xxx",
k8s_node_name="xxxeu-west-1.compute.internal",
k8s_node_uid="54084d95-ecc4-406b-8ac7-11c9a9a6bf57",
k8s_pod_name="xxx-6bzl6",
k8s_pod_uid="ca256889-747f-493d-b262-c8c2ae728f5e",
k8s_replicaset_name="xxx",
namespace="xxx",
node_name="xxx.eu-west-1.compute.internal",
os_type="linux",
otel="true",
pod="xxx-6bzl6",
scrape_endpoint="http-envoy-prom",
scrape_http_scheme="http",
scrape_net_host_name="100.64.179.192",
scrape_net_host_port="15090",
service_instance_id="100.64.179.192:15090",
That said .. what this feels like is some kind of a n issue where the GC is unable to clean up the data when the memory_limiter is tripped. So not an ongoing memory leak, but perhaps just a stuck pointer or something that prevents the data from being collected in some situations?
How about we just process the resource attributes in prometheusremotewrite.FromMetrics instead of pkg/resourcetotelemetry
?
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers
. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Component(s)
exporter/prometheusremotewrite, pkg/resourcetotelemetry
What happened?
Description
We're troubleshooting an issue where _a single
otel-collector-...
pod out of a group begins to turn away metrics because thememorylimiter
is tripped. In this situation, we have dozens or hundreds of clients pushingotlp
metrics (prometheus collected, but over the otlp grpc exporter) to multipleotel-collector-...
pods. The metrics are routed with theloadbalancer
exporter usingrouting_key: resource
.The behavior we see is that one collector suddenly starts running out of memory and being limited .. while the other collectors are using half or even less memory to process the same number of events. Here's graphs of the ingestion and the success rate:
The two dips in the
Percentage of Metrics Accepted by Receiver
graphs are different pods in a StatefulSet. Here's the graph of actual memory usage of these three pods:In the first dip - from 8:30AM->9:30AM, I manually restarted the pod to recover it. It's fine now ... but a few hours later, a different one of the pods becomes overloaded. Grabbing a HEAP dump from the
pprof
endpoint on a "good" and "bad" pod shows some stark differences:Bad Pod Pprof: otel-collector-metrics-processor-collector-0.pb.gz![image](https://github.com/open-telemetry/opentelemetry-collector-contrib/assets/768067/123cd57c-a367-4b29-a87a-c7b9de0cbf03)
Good Pod Pprof: otel-collector-metrics-processor-collector-1.pb.gz![image](https://github.com/open-telemetry/opentelemetry-collector-contrib/assets/768067/c0616539-2df6-4ec7-a49c-8943840cec43)
I should note that this isn't remotely our largest environment - and these pods are handling ~12-15k datapoints/sec, while our larger environments are doing ~20k/sec/pod... so this doesn't feel like a fundamental scale issue. All pods are sized the same across all of our environments.
Collector version
v0.101.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
Log output
No response
Additional context
No response