search

grafana / helm-charts

Apache License 2.0
1.66k stars 2.28k forks source link

Upgrading Loki to HA micro service via helm #356

Open amitrana93 opened 3 years ago

amitrana93 commented 3 years ago

Hi Team, Is there any docs or link from we can upgrade PLG stack to HA cluster via helm? As of right now, only a single Loki stateful pod is running.

amitrana93 commented 3 years ago

??

Whyeasy commented 3 years ago

Hey @amitrana93

As far as I know there isn't really an upgrade document available.

If you are storing your logs within S3 compatible buckets, you can install the charts separately without the loss of data.

To be clear, with stack you mean the loki-stack chart? If so, you can do it gradually. You can for example disable promtail in the loki-stack chart and deploy the promtail chart separately. See if everything works, migrate grafana to the separate chart and etc.

Hopefully this helps 😄

amitrana93 commented 3 years ago

Thanks for the reply @Whyeasy but we need to deploy Loki as a HA cluster. eg. running multiple pods of Loki. Right now we are facing an issue with Loki when we search complex queries or retrieving data for longer intervals.

getting an error:

Screenshot from 2021-04-22 22-32-55

504 gateway timeout.

Whyeasy commented 3 years ago

In this case you should take a look at the loki-distributed chart. This is the micro service deployment of Loki.

The loki chart can't be used for this.

As long as your logs are in S3, you can switch easily. Just point the new Helm chart the same bucket.

amitrana93 commented 3 years ago

Thanks .. it looks great. just one more question. we used to store logs in Dynamodb earlier and now upgrade to s3 with the new version of Loki. Right now we are using both. To search older logs Dynamodb and new one s3. will there is any issue to implement through this chart.

Whyeasy commented 3 years ago

If you keep the same schema and storage config you should be fine. So providing your own config via values.yaml and double check everything 😄