Restart grafana after updating dashboards

canonical / grafana-k8s-operator

This charmed operator automates the operational procedures of running Grafana, an open-source visualization toolkit, on Kubernetes.

https://charmhub.io/grafana-k8s

Apache License 2.0

6 stars 23 forks source link

Restart grafana after updating dashboards #319

Open samuelallan72 opened 3 months ago

samuelallan72 commented 3 months ago

This ensures that dashboard changes are picked up by grafana.

Fixes #303

chanchiwai-ray commented 3 months ago

Did not apply this patch directly, but I tried /charm/bin/pebble restart grafana, and it will force refresh the dashboards.

samuelallan72 commented 3 months ago

Note that we should take a second pass sometime to optimise calls to grafana_restart(), to avoid it being restarted too many times. This will need more investigation of the code to move calls around so that it is only restarted once when necessary, and we don't miss places where it should be restarted.

mmkay commented 2 months ago

Hi @samuelallan72, thank you for your contribution! There seems to be a reload API that might update the dashboards without going through the whole Grafana restart: https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations

Could you have a look if that doesn't fix your issue in a better way?

samuelallan72 commented 2 months ago

@mmkay this may work, but it would involve adding a lot more logic to the charm:

a admin user on grafana for internal use by the charm (the charm currently creates an admin user, but doesn't manage it after that - eg. the user could change the password)
how to handle the case where a user changes the charm's internal admin user
a method to call the grafana api endpoint here

I think this is probably more work than is worth it to fix this bug? Also we seem to restart grafana fairly often on changes; we've just missed restarting in this case. What are your thoughts on this?

samuelallan72 commented 1 month ago

Hi @mmkay , could you take another look at this? Thanks :)

mmkay commented 3 weeks ago

Hi @mmkay , could you take another look at this? Thanks :)

Hey @samuelallan72 - I've rebased the branch to newest main and integration tests seem to be failing in the PR. At first sight it looks like there might be errors around the restart timing so this might require a deeper look. We have the issue in our team's minds so will likely get back to it in near future.

samuelallan72 commented 3 weeks ago

Thanks for looking into it @mmkay ! I don't have much time right now to investigate the tests, but I can try when I do have time if you haven't already. :grin:

Note that there may be a better or more efficient method for deciding when to restart grafana to get it to pick up the dashboard changes. This was just a method that worked for me and seemed a reasonable starting point.