search

pyrra-dev / pyrra

Making SLOs with Prometheus manageable, accessible, and easy to use for everyone!
https://demo.pyrra.dev
Apache License 2.0
1.25k stars 113 forks source link

struggling to configure pyrra with Victoria Metrics. #1258

Closed deanb-everc closed 2 months ago

deanb-everc commented 2 months ago

Hi, my team is evaluating using pyrra for our SLO calculations.

We recently migrated from Prometheus to Victoria Metrics.

I installed the pyrra helm chart on my cluster, and I'm struggling to understand what is the correct prometheusUrl that I should pass in order for pyrra to work.

Currently, I created a test SLO resource, and getting this error in the pyrra operator logs:

level=warn service=objective ts=2024-08-29T13:50:49.891200927Z caller=main.go:1017 msg="failed to query alerts" query="ALERTS{slo=~\".+\"}" err="prometheus query: bad_response: readObjectStart: expect { or n, but found r, error found in #1 byte of ...|remoteAddr:|..., bigger context ...|remoteAddr: \"<REDACTED>, X-Forwarded-For: 1|..."

This error essentially shows that the api call done to Victoria metrics reaches an API that it cannot handle, as this error is the error I get when querying a non-existent API in Victoria Metrics, here's an example to /api/v1/rulez:

remoteAddr: "<REDACTED>, X-Forwarded-For: 10.30.0.62"; requestURI: /api/v1/rulez; unsupported path requested: "/api/v1/rulez"

The issue is that the log from pyrra redacts the requested path, so I'm not sure how to continue debugging this.

I know that the API for Victoria Metrics is largely similar to Prometheus, so here's the endpoint I passed as prometheusUrl: http://vmalert-dev.company.io

The reason I went for this endpoint is because with Victoria Metrics this is the component you should query when trying to get information regarding recording/alerting rules.

I of course made sure that vmalert-dev.company.io works, and /api/v1/rules works (ran a CURL request from within the cluster and got back information).

I went over the main.go file, and couldn't really identify to which endpoints the requests from pyrra arrive to on prometheus.

TL; DR, I'm trying to configure Victoria Metrics as the prometheus endpoint for pyrra and it's failing.

deanb-everc commented 2 months ago

stupidly I didn't look at the logs of Victoria Metrics node. I checked it and saw that the endpoint pyrra is trying to access is /api/v1/query and /api/v1/query_range - those are present on the vmselect component (not vmalert).

it's important to note that this is the path you need to specify when using VictoriaMetrics:

http:///select/0/prometheus

taken from the official docs for Victoria metrics: https://docs.victoriametrics.com/url-examples/#apiv1query