Aggregation keeps increasing

[code-crusher]

Issue

When pushing from multiple/single service, a counter/gauge keeps on aggregating on every push(which makes sense) but metrics like CPU and RAM keeps in increasing. For eg; if first push has process_cpu_system_seconds_total 0.329092 and another one has process_cpu_system_seconds_total 1.210776 from the same service, it gets added which is incorrect CPU value.

Can we reset the vectors upon scrape? I tried below

func (a *Aggregate) HandleRender(c *gin.Context) {
    MetricPushes.Reset()
    contentType := expfmt.Negotiate(c.Request.Header)
    c.Header("Content-Type", string(contentType))
    a.encodeAllMetrics(c.Writer, contentType)

    MetricCountByType.Reset()
    MetricCountByFamily.Reset()
    MetricPushes.Reset()

    fmt.Println("clearing")
}

But did not work.

Would appreciate some help here. I'm not a golang dev so maybe I'm understanding things wrong? TIA!

[SpangleLabs]

Sounds like you want to be pushing those particular metric series to a prometheus push gateway, which exposes the latest value of the metric series, rather than to a prometheus aggregation gateway

[code-crusher]

Thanks for the quick response @SpangleLabs!

Yeah, I probably went down a wrong rabbit hole here. My understanding was this aggregation can merge multiple tasks in microservices, probably something I missed in understanding/implementation.

I went back to the generic push gateway and using labels was able to merge via the PromQL.

Closing this one out.