Align grafana org-based multi-tenancy with loki tenants - Investigation

[Rotfuks]

Motivation

We already have a cluster-based tenant concept in loki. This is historically grown - but not really aligned with our new grafana organisation-based mutli-tenancy concept. This might lead to some confusion and maintenance issues in the future. Aligning the tenancy concepts would definitely reduce the complexity here.

Todo

• [ ] Investigate (PoC) what would change if we align the multi-tenancy concepts of Loki and Grafana
  • Test it out on one installation and see what breaks :)
  • Make sure we can revert the change back, if too much stuff breaks
• [ ] Document the risks and discuss with the team what approach to take

Outcome

• We know what's the risk and decide for an approach to align or not align the multi-tenancy of loki and grafana

[QuentinBisson]

To make this work we need a couple of changes:

• [ ] Push all giant swarm logs (so current ones) to the giantswarm tenant (https://github.com/giantswarm/logging-operator/pull/215)
• [ ] Add support to extract the tenant from a label on the pod log so customers can set their own tenant https://github.com/giantswarm/logging-operator/pull/261
• [ ] See what breaks on big installations
• [ ] Get rid of the multi-tenant proxy for the write path (using an ingress should be enough)
• [ ] Get rid of the old datasource
• [ ] Get rid of the multi-tenant proxy