Loki not receiving/showing logs

[MysterioO]

Image version: ghcr.io/kyverno/policy-reporter:2.20.0

We've defined Loki target and in the policy reporter logs we can see: 1.7222519745660264e+09 info Loki: PUSH OK However, when we go to Loki, I cannot see any logs coming from source: policy reporter. We are also getting notifications in Slack, but not in Loki. I am looking for some ideas how to troubleshoot this issue, if someone had same/similar issues. As I cannot see any errors on Loki side, and kyverno is saying PUSH OK. But logs are not in Loki. 🤷‍♀️

[fjogeleit]

Hey, just to be sure, the source should be policy-reporter with a dash.

Did you overwrite the path in your Loki configuration or do you use the default value? Which Loki version do you running?

[MysterioO]

Yes, I am overwriting it, please see bellow:

loki:
# loki host address
host: "https://gateway.loki.xxxx.xxx/loki"
# list of sources which should send to loki
path: "/api/v1/push"
certificate: "/etc/certs/ca.crt"
sources:
      - kyverno

Loki helm chart is - loki-distributed v0.78.1 .

[fjogeleit]

I got it working by using /api/prom/push as API endpoint instead of /loki/api/v1/push

target:
  loki:
    host: http://loki-loki-distributed-distributor.grafana:3100
    path: "/api/prom/push"
    sources:
      - kyverno

[MysterioO]

My understanding is that "/api/prom/push" is deprecated (though not sure if still available)? I set up push endpoint like in my post above, as I know that it works already for logs that we are pushing over OTEL, see below for OTEL snippet:

  loki:
    endpoint: "https://gateway.loki.xxxx.xx/loki/api/v1/push"

I've set policy-reporter to debug mode and having this output.

2024-07-29 14:00:15 INFO Loki: PUSH OK 2024-07-29 14:00:15 DEBUG Received response: HTTP/2.0 204 No Content Content-Length: 0 Date: Mon, 29 Jul 2024 14:00:15 GMT Strict-Transport-Security: max-age=15724800; includeSubDomains

[fjogeleit]

This endpoint might be deprected but its still available and working. My assumption is that loki/api/v1/push expects a different format which leads to not processing the logs correctly.

I will take a look on this but the current workaround is the usage of the other endpoint.

[MysterioO]

Its working with the "/api/prom/push".