Exporting logs to grafana is not trivial. In order to do that we need a datasource that exports logs. Grafana relies on Grafana Loki [1] to do that. Loki is a system that aggregates, index, stores and serves logs.
However, to retrieve the logs, Loki relies on Grafana Promtail [2] that is an agent that runs on each node.
This PR:
created a Promtail daemonset that scrapes the pod logs. In a future PR, we might want to scrape other system info, container logs, etc.
created a Loki service that aggregates and stores the logs from the Promtail daemonsets. However, the Loki service runs in a single pod, and stores the logs in the local filesystem. In a future PR we might need to tune the storage so we can store more logs, and/or scale up the Loki service.
added a Loki datasource to grafana
After this PR, Grafana should be the ultimate place to inspect logs, metrics, and traces.
grafana_logs_new.webm
Exporting logs to grafana is not trivial. In order to do that we need a datasource that exports logs. Grafana relies on Grafana Loki [1] to do that. Loki is a system that aggregates, index, stores and serves logs.
However, to retrieve the logs, Loki relies on Grafana Promtail [2] that is an agent that runs on each node.
This PR:
After this PR, Grafana should be the ultimate place to inspect logs, metrics, and traces.