Serilog does not work with Grafana Agent due to invalid Loki URL

garethparris commented 1 year ago

Which version of Serilog.Sinks.Grafana.Loki are you using?

v8.1.0

Which version of .NET are you using?

net6.0

Describe the bug

Using the latest version of Grafana Agent 0.34.3 running as a service on my local windows machine, I have several .NET6.0 applications using Serilog Loki Sink sending logs to local URL: http://localhost:3100 and it does not work because it expects the host path to be: http://localhost:3100/api/v1/push instead of http://localhost:3100/loki/api/v1/push.

I was able to discover this by enabling Grafana Agent debug mode and I saw a 404 error being reported in the windows event log: ts=2023-06-30T12:54:39.063912Z component=loki.source.api.loki_push_api level=debug msg="POST /loki/api/v1/push (404) 0s"

Previously I was using Promtail with Serilog Loki Sink and specifying the local URL to be: http://localhost:3100 it was working correctly. It also worked correctly when pushing directly to the Loki server.

I can see that the path is hardcoded in the sink source code:

internal static class LokiRoutesBuilder
{
    private const string LogEntriesEndpoint = "loki/api/v1/push";

    public static string BuildLogsEntriesRoute(string host) => $"{host.TrimEnd('/')}{LogEntriesEndpoint}";
}

To Reproduce

Install Grafana Agent and configure as described here: https://grafana.com/docs/agent/latest/flow/reference/components/loki.source.api/
Enable Grafana Agent debug logging.
Send output from your application using the Serilog Loki Sink to the Grafana Agent.
No data will appear in Loki and there will be an error in the Windows Event Log.

Expected behavior

I expect to see my log output appear in Grafana Loki after being forwarded by Grafana Agent.

Log/SelfLog output or exception with stacktrace

ts=2023-06-30T12:54:39.063912Z component=loki.source.api.loki_push_api level=debug msg="POST /loki/api/v1/push (404) 0s"

Application or code sample, which could be used to reproduce a bug

No response

Additional context

No response

I have read the documentation

[X] I have read Common Problems section in project wiki

mishamyte commented 1 year ago

Hi @prime23-gareth!

Thank you for the issue report. The reason why that path was put into a constants, cause it is strictly defined in Loki HTTP API (Push log entries to Loki).

And that path is supported with Loki standalone deployment, as well as Grafana Cloud.

Looks like Grafana Agent violated that convention, that is not a good one. Probably it is a good point for creating the enhancement about the providing API endpoint path.

garethparris commented 1 year ago

I've raised a bug with Grafana Agent now as a result: https://github.com/grafana/agent/issues/4317

mishamyte commented 1 year ago

I've raised a bug with Grafana Agent now as a result: grafana/agent#4317

Thank you, subscribed. Let's see what would we achieve

mishamyte commented 1 year ago

I think PR in Grafana Agent was closed, so prob with the next release you will have a possibility to setup and work with it correctly

garethparris commented 1 year ago

Yes correct but note their comments, they are trying to get clients to allow changing of the endpoint too. https://github.com/grafana/agent/issues/4317#issuecomment-1622102735

garethparris commented 6 months ago

This is all working for me now so happy to close this.

github-actions[bot] commented 5 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue.

serilog-contrib / serilog-sinks-grafana-loki