search

deis / postgres

A PostgreSQL database used by Deis Workflow.
https://deis.com
MIT License
36 stars 22 forks source link

perform backup on shutdown #164

Closed nailgun closed 7 years ago

nailgun commented 7 years ago

Now backup is performed as background job. It would be nice to perform backup before shutdown, when a container gets TERM or INT signal.

bacongobbler commented 7 years ago

backups are continuously performed with wal-e, which guarantees that a final WAL log will be shipped when SIGTERM is sent. In order to perform a base backup on shutdown, we would have to wait for this final WAL to to be shipped to S3/GCS, then perform the base backup.

As of right now we perform a base backup as soon as the database has restored on first boot.

The problem with performing a base backup on shutdown is that it could potentially go over the 10 second grace period between SIGTERM and SIGKILL. It's generally a good idea to keep the graceful shutdown time to a minimum, which is why only wal-e ships the last WAL rather than doing a full backup.

Does that make sense?

nailgun commented 7 years ago

It does. However doesn't help. Where this grace period is defined? Is it docker or kubernetes specific?

bacongobbler commented 7 years ago

Docker. https://docs.docker.com/engine/reference/commandline/stop/

What are you trying to achieve with this, or what is the issue you're seeing? Perhaps there's an alternative solution. If you're concerned with a long delay between the base backups, you can tune it using BACKUP_FREQUENCY to a shorter interval. I've also bumped the default frequency in #165 to every 4 hours so base backups should occur more often.

nailgun commented 7 years ago

Actually I didn't expected this container will backup itself. But after I've found this, I expected it to backup before termination.

BTW kubernetes defines its own grace period. And there is also pod's PreStop hook which can be used.

bacongobbler commented 7 years ago

I guess we cold add a call to backup in the pre-stop hook. It will wont be a true full backup due to the last WAL log being shipped before shutdown, so there will still be a WAL log to restore from. I'll see what I can do.

nailgun commented 7 years ago

Thank you!