timescale / timescaledb-backup

Other
33 stars 9 forks source link

Move jobs which may cause dump failures & wait on running ones #28

Closed davidkohn88 closed 3 years ago

davidkohn88 commented 3 years ago

Background jobs which perform DDL or change tables in any way can cause issues if they run during a parallel dump, as there can be deadlocks which will cause the dump to be cancelled. We add a co-process which looks for jobs ever 30s, waits for any that are running to finish, and moves any that are scheduled in the next 10 minutes and moves them so that they will occur sometime in the next 15-20 minutes (with some random jitter so we don't get pileups). When finished it reschedules the jobs it moved for sometime in the next 5 minutes or, if the original time is greater than the current time, then for the original time. We introduce a few dump related flags to control this behavior.