Open Ronan-WeScale opened 3 months ago
I tried to deploy observatorium on dev cluster and call loki on other cluster, loki is exposed by nginx ingress.
It seems the api don't set the right Host header on the call, observatorium log :
level=debug name=observatorium ts=2024-08-09T09:26:42.711366336Z caller=proxy.go:61 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 msg="request to upstream" url="https://loki.xxx.com/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000" level=debug name=observatorium ts=2024-08-09T09:26:42.714032285Z caller=instrumentation.go:36 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 proto=HTTP/1.1 method=GET status=404 content= path=/api/logs/v1/xxx/loki/api/v1/query duration=3.310348ms bytes=146
With tcpdump I can catch the header and see the Host isn't my loki address "loki.xxx.com" but my observatorium url (obs.xxx.com), so it's normal the nginx ingress answer 404.
Host: obs.xxx.com User-Agent: Grafana/10.4.3 Accept-Encoding: gzip Authorization: Bearer yyyy Forwarded: for=10.128.4.27;host=obs.xxx.com;proto=https X-Datasource-Uid: e9b1dd3f-49c9-4bff-bdec-f53bd2ee7ea9 X-Forwarded-For: 10.128.4.27 X-Forwarded-Host: obs.xxx.com X-Forwarded-Port: 443 X-Forwarded-Prefix: /api/logs/v1/infra X-Forwarded-Proto: https X-Grafana-Org-Id: 1 X-Id-Token: yyyy
As workaround I had bypass my ingress with metallb to forward traffic directly on loki-gateway svc.
Hi, yes I do this to.
Ronan-WeScale
commented
1 month ago
I tried to deploy observatorium on dev cluster and call loki on other cluster, loki is exposed by nginx ingress.
It seems the api don't set the right Host header on the call, observatorium log :
With tcpdump I can catch the header and see the Host isn't my loki address "loki.xxx.com" but my observatorium url (obs.xxx.com), so it's normal the nginx ingress answer 404.
As workaround I had bypass my ingress with metallb to forward traffic directly on loki-gateway svc.