Duplicate. Loki app: Deploy and set up log forwarding from workload clusters (internal)

cokiengchiara commented 4 years ago

Towards https://github.com/giantswarm/roadmap/issues/175

Tasks tracked in https://github.com/giantswarm/roadmap/issues/202

Outcome:

A new cluster for running loki, logs coming in from external clusters (frontend and operations)

Tasks

[x] upgrade chart to 2.0 and non-demo deploy topology

https://github.com/grafana/loki/issues/2593 https://github.com/unguiculus/loki-helm-chart/

[x] create loki TC in CP and deploy Loki there
[x] figure out how to configure nginx IC to do any form of auth for accepting logs from promtail sources
[x] figure out how to do multi-tenant setup - separating logs of different clusters (if needed)
[x] setup log forwarding from TC clusters

stretch goal

[ ] provide some nice dashboards using logs

Context

From before Tue, 24. November 2020

Details, Background

From @jgsqware Thu, 29. Oct 2020: From Loki 2.0, we can now create alert on query on the logs ingested from loki. That would be really helpful to identify issues in nginx but not only. Yesterday we had issues in calico-kube-controller and calico even if the pods was still running. We had similar behavior with kube-proxy and kiam-server. Would be useful for us to have in place in the near future. It would easier to follow issues during upgrades.

Blocked by / depends on

As of Thu, 29. Oct 2020, from @glitchcrab - we're waiting for the Loki team to move the chart to it's own repo before we can realistically think about using it - lots of stuff will change with this and it should make it easier for us to use the upstream chart with minimal changes. https://github.com/grafana/loki/issues/2593

cokiengchiara commented 3 years ago

@piontec can you write down the progress you made w Loki? 😁

piontec commented 3 years ago

I just made PoC run: the chart is imported to our repo, but not yet good enough to use.No work done on authorization and promtail.

cokiengchiara commented 3 years ago

@piontec Can already make Loki run w AWS. Not good, haven't tried it yet. Still missing promtail and multi-tenant for forwarding logs. Still quite a lot of work.

piontec commented 3 years ago

Closing. Real progress was tracked here: https://github.com/giantswarm/roadmap/issues/202

giantswarm / roadmap