Aiven-Open / pghoard

PostgreSQL® backup and restore service
http://aiven-open.github.io/pghoard/
Apache License 2.0
1.31k stars 96 forks source link

Collect upload metrics during additional basebackup phase #583

Closed nicois closed 1 year ago

nicois commented 1 year ago
Where basic, piped or delta basebackup method is selected,
there is a step where data is being uploaded and the backup is
not yet available, but no activity is sent to the statsd agent.

Register an additional callback to collect the number of bytes
transferred and report it alongside the values from a previous phase.
This helps avoid unnecessary alerts due to basebackups spending time
here.
nicois commented 1 year ago

CI is failing due to a dependency on an untagged rohmu release.

codecov[bot] commented 1 year ago

Codecov Report

Merging #583 (57c1cc9) into main (3448a1b) will increase coverage by 0.01%. The diff coverage is 87.50%.

Additional details and impacted files [![Impacted file tree graph](https://codecov.io/gh/aiven/pghoard/pull/583/graphs/tree.svg?width=650&height=150&src=pr&token=nLr7M7hvCx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven)](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven) ```diff @@ Coverage Diff @@ ## main #583 +/- ## ========================================== + Coverage 91.31% 91.33% +0.01% ========================================== Files 32 32 Lines 4675 4683 +8 ========================================== + Hits 4269 4277 +8 Misses 406 406 ``` | [Impacted Files](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven) | Coverage Δ | | |---|---|---| | [pghoard/transfer.py](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven#diff-cGdob2FyZC90cmFuc2Zlci5weQ==) | `98.68% <83.33%> (+0.01%)` | :arrow_up: | | [pghoard/basebackup/base.py](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven#diff-cGdob2FyZC9iYXNlYmFja3VwL2Jhc2UucHk=) | `92.25% <100.00%> (+0.03%)` | :arrow_up: | | [pghoard/basebackup/chunks.py](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven#diff-cGdob2FyZC9iYXNlYmFja3VwL2NodW5rcy5weQ==) | `96.83% <100.00%> (+0.04%)` | :arrow_up: | | [pghoard/basebackup/delta.py](https://codecov.io/gh/aiven/pghoard/pull/583?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiven#diff-cGdob2FyZC9iYXNlYmFja3VwL2RlbHRhLnB5) | `95.93% <100.00%> (+0.03%)` | :arrow_up: |
nicois commented 1 year ago

@kmichel-aiven coverage has dropped due to no longer attempting to delete non-Path upload objects. I can try to cover this, but I'm not sure it's necessary and the only reason for this change is due to improving the typing, which in turn is related to an unpinned mypy version.