BCDevOps / backup-container

A simple container for a simple backup strategy.
Apache License 2.0
39 stars 56 forks source link

Add retry to the backup process #60

Open WadeBarnes opened 4 years ago

WadeBarnes commented 4 years ago

Occasionally a backup can run into an issue and fail on the first attempt. If attempted again a sort time later it will succeed.

For the sake of robustness add a configurable retry and delay to the backup process so backups are retried if they fail the first time.

Example backup failure:

Backing up 'wallet-indy-cat:5432/agent_indy_cat_wallet' to '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress' ...
--
  | pg_dump: Dumping the contents of table "items" failed: PQgetCopyData() failed.
  | pg_dump: Error message from server: server closed the connection unexpectedly
  | This probably means the server terminated abnormally
  | before or while processing the request.
  | pg_dump: The command was: COPY public.items (id, type, name, value, key) TO stdout;
  | removed '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress'
  | [!!ERROR!!] - Failed to backup wallet-indy-cat:5432/agent_indy_cat_wallet.
  |  
  | Elapsed time: 0h:25m:53s - Status Code: 1

The database server did not terminate. When the backup was retried it succeeded.