Closed thanodnl closed 1 year ago
See also #853 that lead us to using pg_basebackup tar format (maybe even tar.gz) when fetching the data, prior to swapping it in PGDATA. It makes the reasoning about necessary disk space more complex in a way, because now we might still need to have both the “download” area and the “production” area used at the same time for a while.
Given the following in our function pg_basebackup https://github.com/citusdata/pg_auto_failover/blob/d7997ffc3f1209483a37fe7e8ed49fe7a000f664/src/bin/pg_autoctl/pgctl.c#L1280 I would say that https://github.com/citusdata/pg_auto_failover/pull/870 indeed fixed this.
When a failover happens, it does not always succeed to
pg_rewind
the old primary. Instead if has a fallback to recover viapg_basebackup
. This is great!However, once the database becomes bigger in size than 50% of the available diskspace (give or take, inodes could cause other issues) a
pg_basebackup
might not succeed without operator intervention.Instead it would be great if
pg_auto_failover
has an option where, instead of retaining the old database directory, it would delete this directory to ensure enough space is available on the node before initiating apg_basebackup
.Alternatively we could go as far as designing a tristate for this setting:
delete old directory before initiating a restore via pg_basebackup
(maybe even a 4th state where the old data directory is retained till manually deleted - or an other failover happens - so we can perform diagnostics on why pg_rewind failed).
For many installations a
delete old directory before initiating a restore via pg_basebackup
is a very sensible option. If rewind failed pg_auto_failover will copy a fresh copy of the data directory over and configures it as a secondary. This ensures the system always keeps running without an operator needing to ensure enough space is available on the data drive under most circumstances.