Move Grafana to new machine

[fatchat]

Move Grafana to Danswer machine

1. List the configuration files which need to be copied over
2. List any databases / data on the local grafana machine which need to be copied over
3. Copy all resources over
4. Open relevant port
5. Create new EC2 Target
6. Update EC2 LB rule to point to new target
7. Verify setup
8. Shut down old Grafana machine
9. Terminate and delete volume after 15 days

[Ishankoradia]

Grafana and prometheus are on the same machine.

Prometheus - is a open source monitoring toolkit that gathers data (metrics) from multiple sources. It is very good at handling time series data. For migrating prometheus to another machine follow these steps

1. Install prometheus with the correct user (& permissions) according to the instructions here

2. Copy /etc/prometheus/prometheus.yml and /etc/systemd/system/prometheus.service from the old machine to the new

3. Copy the localdb (the timeseries db) at /var/lib/prometheus from the old machine to the new. Use rsync to sync the database directories so we copy incrementally and dont miss out on stats still being pushed while we test & do this. https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories

4. Start and enable the prometheus service to start on boot. If everything is in order , prometheuse server will be running on [localhost](http://localhost:9090)

5. Check if all the datasources/targets are active/up from prometheus dashboard. If not , make sure the right ports are expose & this new machine has access to it

Grafana - is the visualization and analytics platform that allows us to create dashboard of the metrics from prometheus & other sources. For migrating grafana to a new machine follow the steps

1. Install the below version of grafana by following the instructions below

2. Copy the database from the old machine /var/lib/grafana/grafana.db. we are using a sqlite3 db.

3. Verify the setup. Grafana runs on port 3000

[fatchat]

does the prod machine push metrics to this new machine or will the new machine pull from prod

firewalls and dns also need to be updated (which i will do)

[Ishankoradia]

Prometheus pulls metrics from the targets (via http enddpoints). So the new machine will pull from prod.

Yes relevant firewalls/dns will need to be updated.

[Ishankoradia]

Install the postgres exporter in the new machine

[Ishankoradia]

Node exporter version