Pushgateway compatibility with Prometheus push library

[zigmund]

Is your feature request related to a problem? Please describe

VictoriaMetrics accepts pushgateway import via /api/v1/import/prometheus endpoint, but it is incompatible with Prometheus push library:

• HTTP status codes (fixed in #3636)
• VictoriaMetrics should support expfmt.FmtProtoDelim - default protocol format used by Prometheus library.
• ??? Currently stuck at previous point.

Describe the solution you'd like

VictoriaMetrics is fully compatible with Prometheus push library as direct replacement for Pushgateway.

Describe alternatives you've considered

No response

Additional information

No response

[valyala]

The temporary workaround is to configure the Pushgateway client with the expfmt.FmtText format (aka Prometheus text exposition format):

import (
  "github.com/prometheus/client_golang/prometheus/push"
  "github.com/prometheus/common/expfmt"
)

p := push.New(...).Format(expfmt.FmtText)
// use p here

[zigmund]

@valyala

Yes, you're right, but most applications, we use Pushgateway for - are side written, without ability to change protocol format.

[kilingzhang]

I also encountered the same problem. I think this is a common issue that everyone will face. Is there any plan to support it?

[tanguofu]

we have same feature support require. Is there any plan to support it? @valyala thanks

[jsanant]

I tried the above approach but vminsert fails to parse the metric.

Code:

package main

import (
    "fmt"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/push"
    "github.com/prometheus/common/expfmt"
)

func main() {

    ccompletionTime := prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "db_backup_last_completion_timestamp_seconds",
        Help: "The timestamp of the last successful completion of a DB backup.",
    })

    prometheus.MustRegister(completionTime)

    completionTime.SetToCurrentTime()

    p := push.New("http://localhost:8480/insert/0:0/prometheus/api/v1/import/prometheus", "go_operation_aggregator").Format(expfmt.FmtText)

    p.Grouping("instance", "stat_pusher")

    p = p.Collector(completionTime)
    if err := p.Push(); err != nil {
        fmt.Println("Could not push completion time to Pushgateway:", err)
    }
}

Error:

Could not push completion time to Pushgateway: unexpected status code 204 while pushing to http://localhost:8480/insert/0:0/prometheus/api/v1/import/prometheus/metrics/job/go_operation_aggregator/instance/stat_pusher: 

vminsert logs:

{"ts":"2023-08-02T11:21:42.792Z","level":"error","caller":"VictoriaMetrics/lib/protoparser/prometheus/parser.go:231","msg":"cannot unmarshal Prometheus line \"}\": missing value"}
{"ts":"2023-08-02T11:21:42.792Z","level":"error","caller":"VictoriaMetrics/lib/protoparser/prometheus/parser.go:231","msg":"cannot unmarshal Prometheus line \"+db_backup_last_completion_timestamp_seconds\\x12?The timestamp of the last successful completion of a DB backup.\\x18\\x01\\\"\\v\\x12\\t\\t\\xfd\\x86t\\x11\\x8f2\\xd9A\": cannot parse value \"timestamp\": unparsed tail left after parsing float64 from \"timestamp\": \"timestamp\""}
{"ts":"2023-08-02T11:22:03.134Z","level":"warn","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:341","msg":"remoteAddr: \"127.0.0.1:49850\"; requestURI: /insert/3043034633:0/api/v1/import/prometheus/metrics/job/go_operation_aggregator/instance/stat_pusher; unsupported path requested: \"/insert/3043034633:0/api/v1/import/prometheus/metrics/job/go_operation_aggregator/instance/stat_pusher\""}

vminsert-version:

vminsert-20221220-230749-tags-v1.85.3-cluster-0-g30b15111a

Is there something I am missing?

[zekker6]

@jsanant This was fixed in v1.87.0 release:

BUGFIX: vmagent: properly return 200 OK HTTP status code when importing data via Pushgateway protocol. See this issue.