VictoriaMetrics won’t pass health check

[RedL1ghtning]

Greetings!

I have Minio and VictoriaMetrics clusters. Trying set up monitoring in Minio console.

When i set MINIO_PROMETHEUS_URL to http://vm-cluster:8481 - healthcheck work fine, but metrics isn`t work. Victoriametrics logs:

2023-05-23T15:00:18.267Z        warn    VictoriaMetrics/app/vmselect/main.go:277        remoteAddr: "MINIO_IP:PORT"; requestURI: /api/v1/label/disk/values; auth error: cannot parse accountID from "v1": strconv.ParseUint: parsing "v1": invalid syntax

When i sey MINIO_PROMETHEUS_URL to http://vm-cluster:8481/select/0/prometheus - healtcheck isn`t work with error "Prometheus URL is unreachable". Victoriametrics logs:

2023-05-23T15:09:13.353Z        warn    VictoriaMetrics/lib/httpserver/httpserver.go:343        remoteAddr: "MINIO_IP:PORT"; requestURI: /select/0/prometheus/-/healthy; unsupported path requested: "/select/0/prometheus/-/healthy"

In this lines addef fall back mechanism for similar problems, but something goes wrong: https://github.com/minio/console/blob/920fc7d937d25dfa466461d238d23df4e6ac757b/restapi/admin_info.go#L987-L1011

Seems like some fuctions didn`t use testPrometheusURL inside

[cesnietor]

@cniackz will try to reproduce.

[PolinaVens]

Is there any update on the issue? I have a similar problem in MinIO VERSION 2023-10-07T15:07:38Z

[ngc6166]

Hi! I have the same in RELEASE.2023-11-20T22-40-07Z.

[vchirikov]

Same with RELEASE.2023-12-23T07-19-11Z@sha256:47d5b01cc3020efa39aac112a7cbc919af5fb18f9ebc63bda31683590715c42b

[rpardini]

Same with DEVELOPMENT.2023-12-23T07-19-11Z (as provided by Bitnami's minio Helm Chart). References https://github.com/minio/console/issues/1373 and https://github.com/minio/console/pull/1431

[dvaldivia]

This is fixed now, please reopen if it keeps happening

[DANic-git]

Reproduce in RELEASE.2024-02-24T17-11-14Z

[autokilla47]

Reproduce in RELEASE.2024-05-10T01-41-38Z

[deepminimal]

Reproduce in RELEASE.2024-08-03T04-33-23Z

[cniackz]

First of all, this isn’t Prometheus, and VictoriaMetrics does not use scrape_configs in the same way Prometheus does. It is also expected that the health check will fail for http://vm-cluster:8481/select/0/prometheus, as including select/0/prometheus in the URL will not match the /-/healthy endpoint. This is reflected in the following changelog entry:

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833

./docs/changelog/CHANGELOG_2021.md: * FEATURE: Expose `/-/healthy` and `/-/ready` endpoints similar to Prometheus. This improvement facilitates integration with third-party solutions that rely on these endpoints. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833).

When I last tested this feature using this guide, I encountered no issues, although I haven’t tested VictoriaMetrics myself.

If PR #956 has broken this feature or if VictoriaMetrics has changed, it would be helpful to identify which commit caused the issue or how VictoriaMetrics is configured. This seems to be more of an integration issue with VictoriaMetrics rather than a problem with Prometheus configuration. We need to discuss internally whether we should continue supporting VictoriaMetrics integration.

[ramondeklein]

VictoriaMetrics doesn’t seem to implement the Prometheus health-checks. We support Prometheus, but don’t support 99% compatible products.

I think it would make more sense to request VictoriaMetrics to support the missing endpoint. It claims to be Prometheus compatible, so it should support it.