VictoriaMetrics / metrics

Lightweight alternative to github.com/prometheus/client_golang
MIT License
526 stars 59 forks source link

Introduce a `Close` method that pushes metrics one last time. #38

Closed bubunyo closed 9 months ago

bubunyo commented 1 year ago

When you use the InitPush method to push metrics from a job, not all metrics get pushed.

This specifically has to do with metrics that are pushed just before a job is terminated. An example of such a metric could be how long a job took to process. Since by the time this metric is pushed, there is never a last metric push, it never gets sent to a cluster.

A solution could be to introduce a Close method perhaps compatible with io.Closer to push all metrics one last time. before a job is killed.

salarali commented 1 year ago

Any updates on this? I am basically looking for a way to unregister a set, hence stopping the initial InitPush I called for that set.

hagen1778 commented 1 year ago

Thanks for the feature request! I think a right way to implement it would be to make InitPush to return an object. The object could satisfy Closer interface and call of Close method should gracefully stop and drain all background workers for this specific remote-write URL. This would satisfy both requests: first and second.

valyala commented 9 months ago

FYI, the commit fd258897112d01ac9a2b7925a50d52c3045a4e49 adds PushMetrics() function, which allows force pushing metrics at the time when the PushMetrics() is called. The commit bd3cd7b6ff35c219e8d4e0593cb53a071dbe1e86 adds ability to cancel periodic push by canceling the context passed to InitPushWithOptions(). So now it is possible to cancel periodic push and to send metrics last time with the following code:

s := metrics.NewSet()
// add metrics to s here

ctx, cancel := context.WithCancel(context.Background())
if err := s.InitPushWithOptions(ctx, pushURL, pushInterval, nil); err != nil {
  panic(err)
}

// do some work
// ...

// stop periodic push
cancel()

// Send metrics last time before exit
if err := s.PushMetrics(context.Background(), pushURL, nil); err != nil {
  panic(err)
}

These commits are available starting from the tag v1.27.0. Closing the feature request as done.