tgross
commented
7 years ago Would it be possible to have the primary database (attempt to) take a snapshot of the DB when it is restarted or shut down?
You could add this as a preStop field in the ContainerPilot configuration. python /usr/local/bin/manage.py write_snapshot would bypass the backup TTL check. This only protects you in the case of a shutdown or restart -- an unexpected crash leaves you in the same situation.
This time, it decided to restart all of the consul instances, the mysql instance, and the foo instance.
I think you want the --no-recreate flag in that case. At Joyent we use Compose for managing our own website (the front web page, not the portal or the datacenter!) and when we want to replace instances we do a rolling deploy by scaling down one instance and replacing it with another.
Would it be possible to have the primary database (attempt to) take a snapshot of the DB when it is restarted or shut down?
I had rebuilt a custom docker container (not the mysql one) and wanted to update it in the runtime. What I normally do is do a
docker-compose build foo, thendocker-compose up -d footo have it replaced in the project environment. Usually, only that one instance (foo) is recreated.This time, it decided to restart all of the consul instances, the mysql instance, and the foo instance. I think this would have been OK, but for some reason the consul raft sank in the process triggering an unfortunate chain reaction resulting in the loss of the mysql data after the last backup.
If mysql could take one last snapshot before going away, I could fix the situation manually. Without a last backup, all of the data that was in the container when it went down is gone.