search

aws / amazon-managed-grafana-roadmap

Amazon Managed Grafana Roadmap
Other
57 stars 4 forks source link

Enable Live Tailing for Loki Datasources #66

Open aristosvo opened 8 months ago

aristosvo commented 8 months ago

Live tailing is mentioned here: https://docs.aws.amazon.com/grafana/latest/userguide/v9-explore-logs.htm

Logs sample If the selected data source implements logs sample, and supports both log and metric queries, then for metric queries you will be able to automatically see samples of log lines that contributed to visualized metrics. This feature is currently supported by Loki data sources.

Live tailing

Use the Live tailing feature to see real-time logs on supported data sources.

Select the Live button in the Explore toolbar to switch to Live tail view.

While in Live tail view new logs will come from the bottom of the screen and will have fading contrasting background so you can keep track of what is new. Select the Pause button or scroll the logs view to pause the Live tailing and explore previous logs without interruption. Select Resume button to resume the Live tailing or select Stop button to exit Live tailing and go back to standard Explore view.

But in practice it doesn't work at all, as it seems for now. With logcli I am able to verify that the Loki tailing works without Grafana.

It would be really great to have that available!

Related issue: https://github.com/aws/amazon-managed-grafana-roadmap/issues/34

achille-roussel commented 8 months ago

I am hitting the same issue, logcli works as expected but attempting to tailing loki logs from Amazon managed Grafana fails when establishing the WebSocket connection between the browsers and Grafana.

This is the response received by the browser in this case:

> GET /api/datasources/proxy/uid/YHg4jD4Iz/loki/api/v1/tail?query=%7Bservice%3D%22prometheus%22%7D%20%7C%3D%20%60%60 HTTP/1.1
...
< HTTP/1.1 403 Forbidden
< Date: Mon, 06 Nov 2023 18:47:10 GMT
< Content-Length: 0
< Connection: keep-alive
< vary: Accept-Encoding
<

I've searched CloudTrail logs for this error but could not find anything. I also confirmed that the loki backend does not receive a WebSocket connection, so the issue appears to be in the Grafana configuration.