Default Loki ServiceMonitor is trying to scrape gateways non-existent /metrics endpoint

[den-is]

Describe the bug Default Loki's ServiceMonitor is making Prometheus to scrape loki-gateway's non-existent /metrics endpoint

To Reproduce Steps to reproduce the behavior:

1. Install stand loki installation deploymentMode: SimpleScalable, with gateway enabled
2. Install simple kube-prometheus-stack
3. Check ServiceMonitor for loki
4. Check Prometheus targets for loki - there will be

Expected behavior No red/down targets in prometheus.
No 404 errors in loki-gateway logs.
Ignore loki-gateway in ServiceMonitor or fix /metrics endpoint for loki-gateway.

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm, official and latest loki chart

Screenshots, Promtail config, or terminal output

• loki-gateway log message

  loki-gateway-56d5d7df7b-m6mqc nginx 10.244.45.238 - - [12/Jun/2024:07:25:31 +0000]  404 "GET /metrics HTTP/1.1" 153 "-" "Prometheus/2.52.0" "-"

• curl output from loki-gateway

  
  curl -iv http://localhost:8080/metrics

• Host localhost:8080 was resolved.

• IPv6: ::1

• IPv4: 127.0.0.1

• Trying [::1]:8080...

• Connected to localhost (::1) port 8080

  GET /metrics HTTP/1.1 Host: localhost:8080 User-Agent: curl/8.5.0 Accept: /

  < HTTP/1.1 404 Not Found HTTP/1.1 404 Not Found < Server: nginx/1.24.0 Server: nginx/1.24.0 < Date: Wed, 12 Jun 2024 08:04:18 GMT Date: Wed, 12 Jun 2024 08:04:18 GMT < Content-Type: text/html Content-Type: text/html < Content-Length: 153 Content-Length: 153 < Connection: keep-alive Connection: keep-alive

<

404 Not Found

---

nginx/1.24.0

• Connection #0 to host localhost left intact

• prometheus failed targets

  

[strowi]

Ran into the same problem...

The serviceMonitor scrapes all "http-metrics"-ports and the gateway has mistakenly (?) been namend the same instead of {{ include "loki.gatewayFullname" . }} (can't speak to the enterprise-version though).

I fixed it here

[den-is]

@strowi thanks! I am not 100% sure that this is exactly authentic fix for my exact issue. Just my IMHO. I don't like when ports are not names after generic protocol names. (http, grpc, etc with whatever prefixes/suffixes)

But it 100% fits the original port name issue https://github.com/grafana/loki/issues/12963 and probably your naming strategy can be discussed there.

[strowi]

@den-is totally agree! And i didn't know about the original issue. I just had to quick-fix it in out deployment and used a the service-name/var from the ingress-definition ;)

[Skaronator]

I created an PR to fix this. As a temporary workaround you could just set these label on the service:

gateway:
  service:
    labels:
      prometheus.io/service-monitor: "false"

[TimJongerius]

We've just upgraded to the latest Loki version (was a pain by the way). The default service monitor still tries to scrape metrics from the non existent metrics endpoint of the gateway.

Can you please provide the endpoint or update the servicemonitor?

This issue is open since Jun 12. What is keeping you from fixing it?