search

Waziup / WaziCloud

WAZIUP Cloud and local platform
31 stars 28 forks source link

Backup/restore #180

Closed cdupont closed 6 years ago

cdupont commented 6 years ago

I open this issue to discuss the backup/restore of the platform. There are 3 databases:

In this issue we'll take the example of copying all data from the online instance to a local instance of the platform. The goal is to create locally a replica of the online platform, with all users, sensors and assets.

First start Waziup locally:

docker-compose up

Keycloak

mysqldump can be used to backup keycloak.

mysqldump -h dev.waziup.io -P 3306 -u root -proot_password --all-databases > /var/backups/mysqlbackups/`date +"%m-%d-%y"`

Restore the data:

mysql -u root -proot_password <  /var/backups/mysqlbackups/`date +"%m-%d-%y"`

Alternatively, it's possible to export all realms using keycloak command line:

ssh ec2-user@dev.waziup.io
//get the keycloak container id and log in:
docker ps
sudo docker exec -it 9b7afcbfd7fe /bin/bash
//Export keycloak database
./keycloak/bin/standalone.sh -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=/opt/jboss/keycloak/standalone/data/export-`date +"%m-%d-%y"` -Djboss.http.port=8888 -Djboss.https.port=9999 -Djboss.management.http.port=7777
//Interrupt with Ctrl-C and exit twice: Ctrl-D Ctrl-D
//copy the export files:
scp -r ec2-user@dev.waziup.io:/data/keycloak/export-`date +"%m-%d-%y"` .

At this stage you should be able to import users and realms in Keycloak. Open the admin console: http://localhost:8080/auth/ and add the export files in the import menu.

Mongo

This command archives the remote mongo database:

mongodump --host dev.waziup.io --port 27017 --out /var/backups/mongobackups/`date +"%m-%d-%y"`

This command restores it in the local instance:

mongorestore /var/backups/mongobackups/`date +"%m-%d-%y"`

Elasticsearch

Dump the database from remote:

elasticdump --input=http://dev.waziup.io:9200/ --output=els-`date +"%m-%d-%y"` --type=data --headers='{"Content-Type": "application/json"}'

Insert the data locally:

elasticdump --output=http://localhost:9200/ --input=els-`date +"%m-%d-%y"` --type=data
cdupont commented 6 years ago

Crontab on the VM:

30 1 * * * sudo mongodump --out /mnt/S3/backups/mongobackups/`date +"%m-%d-%y"`
45 1 * * * sudo mysqldump -h 127.0.0.1 -u root -proot_password --all-databases > /mnt/S3/backups/mysqlbackups/`date +"%m-%d-%y"`

S3 storage was used following this tutorial: https://cloudkul.com/blog/mounting-s3-bucket-linux-ec2-instance/