Closed nailgun closed 7 years ago
backups are continuously performed with wal-e, which guarantees that a final WAL log will be shipped when SIGTERM is sent. In order to perform a base backup on shutdown, we would have to wait for this final WAL to to be shipped to S3/GCS, then perform the base backup.
As of right now we perform a base backup as soon as the database has restored on first boot.
The problem with performing a base backup on shutdown is that it could potentially go over the 10 second grace period between SIGTERM and SIGKILL. It's generally a good idea to keep the graceful shutdown time to a minimum, which is why only wal-e ships the last WAL rather than doing a full backup.
Does that make sense?
It does. However doesn't help. Where this grace period is defined? Is it docker or kubernetes specific?
Docker. https://docs.docker.com/engine/reference/commandline/stop/
What are you trying to achieve with this, or what is the issue you're seeing? Perhaps there's an alternative solution. If you're concerned with a long delay between the base backups, you can tune it using BACKUP_FREQUENCY to a shorter interval. I've also bumped the default frequency in #165 to every 4 hours so base backups should occur more often.
Actually I didn't expected this container will backup itself. But after I've found this, I expected it to backup before termination.
BTW kubernetes defines its own grace period. And there is also pod's PreStop hook which can be used.
I guess we cold add a call to backup in the pre-stop hook. It will wont be a true full backup due to the last WAL log being shipped before shutdown, so there will still be a WAL log to restore from. I'll see what I can do.
Thank you!
Now backup is performed as background job. It would be nice to perform backup before shutdown, when a container gets TERM or INT signal.