Missing runtime metrics

morningspace commented 2 weeks ago

Description

Some runtime metrics, e.g.: memStats.Alloc, memStats.Sys, are missing.

Steps To Reproduce

By checking the code for deprecated runtime, we found there're no runtime metrics for memStats.Alloc, memStats.Sys.
Also in the new runtime metrics implementation, the code to collect the metrics is different. Then, what would be the equivalents to the missing runtime metrics in the new implementation?

Expected behavior

Can the missing runtime metrics be added? If a PR is allowed, we'd be happy to contribute.

ysomad commented 4 days ago

any updates?

dmathieu commented 4 days ago

The non-deprecated runtime uses runtime/metrics to retrieve metrics data. The metrics we exposed are documented in semantic conventions.

It looks like you're looking for go.memory.allocated and go.memory.used.

The current runtime does expose those two metrics on its latest version, even though it uses the runtime/metrics package, not runtime.MemStats.

morningspace commented 3 days ago

Thanks @dmathieu for your reply!

It was mentioned in semantic conventions that go.memory.allocated is the memory allocated to the heap by the application, while in runtime.memStats, it has both memStats.HeapAlloc and memStats.Alloc available. So, I assume go.memory.allocated maps to memStats.HeapAlloc, not memStats.Alloc.
For memStats.Sys, that's the memory obtained from system. When I check the code in non-deprected runtime, I found it actually checks go heap memory defined here, which I'm not sure if it is identical to memStats.Sys.

dmathieu commented 3 days ago

cc @dashpole who led this.

dashpole commented 3 days ago

Based on https://github.com/prometheus/client_golang/blob/76b74e25d5660965000a74cf2e918c217ed76da2/prometheus/go_collector.go#L26, this is the mapping from memStats to go runtime metrics:

memStats.Alloc is the same as the /memory/classes/heap/objects:bytes runtime metric.
memStats.Sys is the same as the /memory/classes/total:bytes runtime metric.

/memory/classes/total:bytes isn't useful on its own, as it includes released memory. For the new runtime metrics, we provide the go.memory.used metric, which excludes released memory.

You can see https://github.com/golang/go/issues/67120 for why we don't provide live + unswept heap memory via /memory/classes/heap/objects:bytes:

live+unswept heap memory isn't a terribly useful metric since it tends to be noisy and misleading, subject to sweep scheduling nuances. The heap goal is a much more reliable measure of total heap footprint.

In keeping with that, we would recommend using the go.memory.gc.goal metric, which measures the heap goal.

open-telemetry / opentelemetry-go-contrib