distributed-system-analysis / pbench

A benchmarking and performance analysis framework
http://distributed-system-analysis.github.io/pbench/
GNU General Public License v3.0
188 stars 108 forks source link

Implement backup for the Server #3553

Closed webbnh closed 1 year ago

webbnh commented 1 year ago

Lost in the v0.69 -> v1.0 transition was the mechanism for making backup copies of result tarballs loaded on to the Server. This gap is currently covered by the fact that the "pass-through" server is v0.69-based and so it makes a backup of the result before pushing it to the "index" server. As things stand, once we upgrade the "index" server to a full-blown v1.0 and have users push results to it directly, we won't have any provision for creating backups. This PR closes that gap.

This change adds a pbench-backup-dir configuration option which points to a directory to receive copies of result tarballs. (This directory should be placed on a file system different from the archive directory (preferably on different hardware), although this can easily be effected via a mount point.) This change adds two functions to the IntakeBase class which encapsulate the details of creating a backup copy and of removing it in the event of an upload failure. Beyond that, the change is just to call the backup function and to declare the removal function as a cleanup handler during the intake process. The rest of the changes are revisions to the unit tests for the impact of the configuration and backup changes along with two new unit tests for the two new functions.

The PR is structured as a series of commits for your reviewing pleasure:

PBENCH-1259

webbnh commented 1 year ago

I've squashed today's commits together (because some of them were a tangent), so you may have to re-review a couple of lines (but, mostly, I got rid of changes).

But, you might want to hold off until the unit tests pass....