Mimir should implement `api/v1/status/(config|flags)` endpoints

[upodroid]

Describe the bug

Mimir doesn't implement the full Prometheus API. Kiali uses some of these endpoints to check if Prometheus is running.

https://github.com/kiali/kiali/blob/master/prometheus/client.go (Look for GetConfiguration and GetFlags methods)

A quick and dirty solution is to add additional paths to the nginx proxy to unblock this. https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml#L2049

https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/ https://prometheus.io/docs/prometheus/latest/querying/api/#config

 REDACTED  MCW0CDP3YY  ~  SIGINT  $  k logs kiali-78b56f47c4-5hthv -c istio-proxy  -f | grep prometheus | grep status
W1027 21:09:36.780650   36373 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
[2022-10-27T19:59:38.097Z] "GET /prometheus/api/v1/status/config HTTP/1.1" 404 - via_upstream - "-" 0 19 16 16 "-" "Go-http-client/1.1" "61f67486-9272-4560-8102-e1e99871a642" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:39086 10.201.31.154:80 10.201.82.91:35842 - default
[2022-10-27T19:59:38.114Z] "GET /prometheus/api/v1/status/flags HTTP/1.1" 404 - via_upstream - "-" 0 19 3 3 "-" "Go-http-client/1.1" "d32465b9-6026-4b83-8d04-cb3a5ddf0279" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:39086 10.201.31.154:80 10.201.82.91:35842 - default
[2022-10-27T20:00:22.494Z] "GET /prometheus/api/v1/status/config HTTP/1.1" 404 - via_upstream - "-" 0 19 6 6 "-" "Go-http-client/1.1" "5dbcefbb-1189-4ae3-b3f9-41509060e6f4" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:38962 10.201.31.154:80 10.201.82.91:46244 - default
[2022-10-27T20:00:22.501Z] "GET /prometheus/api/v1/status/flags HTTP/1.1" 404 - via_upstream - "-" 0 19 2 2 "-" "Go-http-client/1.1" "c4efc34a-5811-45ad-b1d8-8e8cde7ff7f2" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:38962 10.201.31.154:80 10.201.82.91:46244 - default
[2022-10-27T20:07:36.440Z] "GET /prometheus/api/v1/status/config HTTP/1.1" 404 - via_upstream - "-" 0 19 6 6 "-" "Go-http-client/1.1" "ad6f2102-0392-40dd-b671-500965e56469" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:60618 10.201.31.154:80 10.201.82.91:47942 - default
[2022-10-27T20:07:36.447Z] "GET /prometheus/api/v1/status/flags HTTP/1.1" 404 - via_upstream - "-" 0 19 3 2 "-" "Go-http-client/1.1" "97a98f69-b770-4f58-b29c-9b748dbacee3" "monitoring-mimir-nginx.monitoring:80" "10.201.80.29:8080" outbound|80||monitoring-mimir-nginx.monitoring.svc.cluster.local 10.201.82.91:60618 10.201.31.154:80 10.201.82.91:47942 - default
^C
 REDACTED  MCW0CDP3YY  ~  SIGINT  $  k logs kiali-78b56f47c4-5hthv | grep 404
W1027 21:09:49.574445   36387 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
2022-10-27T19:59:38Z ERR Failed to fetch Prometheus configuration: client_error: client error: 404
2022-10-27T19:59:38Z ERR Failed to fetch Prometheus flags: client_error: client error: 404
2022-10-27T20:00:22Z ERR Failed to fetch Prometheus configuration: client_error: client error: 404
2022-10-27T20:00:22Z ERR Failed to fetch Prometheus flags: client_error: client error: 404
2022-10-27T20:07:36Z ERR Failed to fetch Prometheus configuration: client_error: client error: 404
2022-10-27T20:07:36Z ERR Failed to fetch Prometheus flags: client_error: client error: 404

# Real "prometheus" server
 maha4472  MCW0CDP3YY  ~  $  curl -s localhost:9090/api/v1/status/config | jq .status
"success"
 maha4472  MCW0CDP3YY  ~  $  curl -s localhost:9090/api/v1/status/flags | jq .status
"success"

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version) 4.0.0-weekly.209
2. Install this helm chart below.

Environment

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop] GKE 1.23
• Deployment tool: [e.g., helm, jsonnet] helm https://github.com/borg-land/k8s-demo/blob/dev/monitoring/values.yaml

[pracucci]

I think it's a good idea supporting /api/v1/status/config and /api/v1/status/flags. Willing to give it a try? We can offer some guidance.

[upodroid]

I'll raise a PR for it early next week. Can you leave some pointers here?

Thanks

[pracucci]

@upodroid Sorry I missed your message. Are you still up for it? If so, I can start typing some suggestions.

[upodroid]

I haven't started it yet.

@hi-rustin You can work on this.

Thanks

[Rustin170506]

@hi-rustin You can work on this.

Thanks! I'll try to add it.

[56quarters]

Completed via #3596

[Rustin170506]

Completed via #3596

Oops! Probably not yet. Do we still need to add api/v1/status/flags?

[Hronom]

I'm still getting error "Prometheus is unreachable", according to access logs issue is next:

127.0.0.6 - - [03/Jul/2023:00:53:11 +0000]  404 "GET /prometheus HTTP/1.1" 19 "-" "Go-http-client/1.1" "-"
127.0.0.6 - - [03/Jul/2023:00:53:11 +0000]  404 "GET /prometheus HTTP/1.1" 19 "-" "Go-http-client/1.1" "-"

Seems like it's fail because endpoint http://localhost:42983/prometheus returns 404.

I able to add workaround by setting this in Kiali config:

      external_services:
        prometheus:
          url: "http://mimir-nginx.mimir.svc.cluster.local:80/prometheus"
          health_check_url: "http://mimir-nginx.mimir.svc.cluster.local:80/"