search

grafana / helm-charts

Apache License 2.0
1.67k stars 2.28k forks source link

Migration from loki-stack to loki helm chart - Seeking Advice on Smooth Transition #3393

Open vitas-pm opened 2 weeks ago

vitas-pm commented 2 weeks ago

Hi Grafana Community,

I'm in the process of migrating from the loki-stack helm chart to the regular Loki chart, as it seems the loki-stack chart is no longer maintained. While I understand that loki-stack was community-supported, I was a bit surprised to see it still hosted on GitHub without clear deprecation notices or guidance for migrating. I hope that future transitions can include clearer notices or migration guides to help users navigate these changes smoothly.

The loki-stack chart previously deployed a standalone version of Loki, which I believe is similar to the current SingleBinary mode. I have a few questions about how best to handle the migration to the SimpleScalable deployment (if possible), ideally without losing existing logs.

  1. About the Migration Guide: Can the official migration guide be used in this case? Some steps mentioned may not be directly configurable in the loki-stack chart, but I assume they could be done by editing Kubernetes resources directly. Would following this guide ensure that my old logs are properly transferred to the new Loki storage?

  2. Storage Configuration: If I want to deploy the new Loki instance in SingleBinary mode, is there a way to reuse the old PVC for persistence? Additionally, is cloud storage (like S3) a requirement now? The previous setup worked well with just persistent volumes. I understand that cloud storage may support the SimpleScalable setup to allow read and write deployments to access the same storage, but I’m curious why the new helm chart still includes PVC persistence options for the read and write components if cloud storage is already used. Is this intended for caching, or is there another purpose?

  3. S3 Credentials: Lastly, with regard to S3 storage, I noticed there isn't an option to retrieve access and secret keys from a Kubernetes secret, which would avoid storing them in plain text in the helm chart. Wouldn’t this be better practice?

I hope my questions are clear, and I appreciate any insights you can provide. Thank you for your time and help!

Best regards

CarlQLange commented 4 days ago

I too was surprised to find the loki-stack chart unmaintained, but with no deprecation notices etc. A guide for migrating would be incredibly helpful!

vitas-pm commented 4 days ago

For anyone facing similar issues, here’s the workaround I used to migrate smoothly:

  1. Deploy New Promtail and Loki: Start by deploying the new Promtail and Loki instances using the latest helm chart. To avoid conflicts, deploy them in the same namespace as your existing setup but use a nameOverride to give them unique names.

  2. Update the Old loki-stack Chart: Modify the old loki-stack deployment to remove the Promtail component, so it doesn’t interfere with the new setup.

  3. Update Grafana’s Data Sources: If you’re using Grafana:

    • Edit the datasource ConfigMap so the default Loki datasource points to the new Loki gateway URL.
    • Add your old Loki instance as a new datasource (e.g., loki-old) to allow querying historical logs from Grafana.

  4. Decommission the Old Loki: Once the old Loki instance has completed its log rotation and is no longer needed, you can safely delete it. From this point, you’ll be fully transitioned to the new setup.

I still wish though there were an actual solution provided by Grafana.