search

Aiven-Open / pghoard

PostgreSQL® backup and restore service
http://aiven-open.github.io/pghoard/
Apache License 2.0
1.31k stars 96 forks source link

Add progress based basebackup metrics #615

Closed sebinsunny closed 6 months ago

sebinsunny commented 7 months ago

This PR improves the monitoring of pg basebackups. During a backup, it regularly checks how much data has been uploaded and compares this to the last recorded amount in a persisted progress file. If the upload is progressing, it updates the record with the new data and current time. If the backup has not advanced compared to the previous value, it reports the time elapsed since the last progress and emits stalled metrics. Once a backup is complete, the record is reset.

[SRE-7631]

codecov-commenter commented 7 months ago

Codecov Report

Attention: Patch coverage is 70.10309% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 90.95%. Comparing base (2b2ea98) to head (6e00a1c).

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615/graphs/tree.svg?width=650&height=150&src=pr&token=nLr7M7hvCx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open)](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open) ```diff @@ Coverage Diff @@ ## main #615 +/- ## ========================================== - Coverage 91.10% 90.95% -0.16% ========================================== Files 32 32 Lines 4823 4918 +95 ========================================== + Hits 4394 4473 +79 - Misses 429 445 +16 ``` | [Files](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open) | Coverage Δ | | |---|---|---| | [pghoard/basebackup/base.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9iYXNlYmFja3VwL2Jhc2UucHk=) | `92.25% <100.00%> (+0.03%)` | :arrow_up: | | [pghoard/transfer.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC90cmFuc2Zlci5weQ==) | `95.78% <64.70%> (-0.09%)` | :arrow_down: | | [pghoard/common.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9jb21tb24ucHk=) | `90.96% <80.35%> (-2.25%)` | :arrow_down: | | [pghoard/basebackup/delta.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9iYXNlYmFja3VwL2RlbHRhLnB5) | `91.20% <45.45%> (-4.05%)` | :arrow_down: | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/615/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open)