Closed MysterioO closed 1 month ago
Hey, just to be sure, the source should be policy-reporter
with a dash.
Did you overwrite the path
in your Loki configuration or do you use the default value?
Which Loki version do you running?
Yes, I am overwriting it, please see bellow:
loki: # loki host address host: "https://gateway.loki.xxxx.xxx/loki" # list of sources which should send to loki path: "/api/v1/push" certificate: "/etc/certs/ca.crt" sources: - kyverno
Loki helm chart is - loki-distributed v0.78.1 .
I got it working by using /api/prom/push
as API endpoint instead of /loki/api/v1/push
target:
loki:
host: http://loki-loki-distributed-distributor.grafana:3100
path: "/api/prom/push"
sources:
- kyverno
My understanding is that "/api/prom/push"
is deprecated (though not sure if still available)?
I set up push endpoint like in my post above, as I know that it works already for logs that we are pushing over OTEL, see below for OTEL snippet:
loki: endpoint: "https://gateway.loki.xxxx.xx/loki/api/v1/push"
I've set policy-reporter to debug mode and having this output.
2024-07-29 14:00:15 INFO Loki: PUSH OK 2024-07-29 14:00:15 DEBUG Received response: HTTP/2.0 204 No Content Content-Length: 0 Date: Mon, 29 Jul 2024 14:00:15 GMT Strict-Transport-Security: max-age=15724800; includeSubDomains
This endpoint might be deprected but its still available and working. My assumption is that loki/api/v1/push
expects a different format which leads to not processing the logs correctly.
I will take a look on this but the current workaround is the usage of the other endpoint.
Its working with the "/api/prom/push".
Image version: ghcr.io/kyverno/policy-reporter:2.20.0
We've defined Loki target and in the policy reporter logs we can see:
1.7222519745660264e+09 info Loki: PUSH OK
However, when we go to Loki, I cannot see any logs coming from source: policy reporter. We are also getting notifications in Slack, but not in Loki. I am looking for some ideas how to troubleshoot this issue, if someone had same/similar issues. As I cannot see any errors on Loki side, and kyverno is saying PUSH OK. But logs are not in Loki. 🤷♀️