Pghoard restore_command optimization

[egor-voynov-aiven]

Changes:

• Added a new DownloadResultsProcessor thread, which validates downloaded files and saves them to the target: pg_wal/<wal_name>.tmp -> pg_wal/<wal_name>.pghoard.prefetch. Files with "prefetch" suffix can be copied to the destination without extra checks now.
• Renaming prefetched WAL to target is carried out directly in the pghoard_postgres_command_go
• changed items type for "pending_download_ops" dict from "dict" to "dataclass" (PendingDownloadOp)

[BF-2247]

We observed that the startup process was only using 60 to 80% of a CPU. This means that we’re not supplying WAL files fast enough or that the restore_command takes too long to complete. The idea would be to bypass the HTTP-call needed to check if we have a WAL file present from the restore_command to the pghoard main process.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.68116% with 17 lines in your changes are missing coverage. Please review.

Project coverage is 90.79%. Comparing base (5505b86) to head (30eb4e5).

Additional details and impacted files

[](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open) ```diff @@ Coverage Diff @@ ## main #618 +/- ## ========================================== - Coverage 91.01% 90.79% -0.22% ========================================== Files 31 31 Lines 4917 4954 +37 ========================================== + Hits 4475 4498 +23 - Misses 442 456 +14 ``` | [Files](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open) | Coverage Δ | | |---|---|---| | [pghoard/common.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?src=pr&el=tree&filepath=pghoard%2Fcommon.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9jb21tb24ucHk=) | `90.96% <100.00%> (ø)` | | | [pghoard/pghoard.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?src=pr&el=tree&filepath=pghoard%2Fpghoard.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9wZ2hvYXJkLnB5) | `85.66% <100.00%> (+0.08%)` | :arrow_up: | | [pghoard/postgres\_command.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?src=pr&el=tree&filepath=pghoard%2Fpostgres_command.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC9wb3N0Z3Jlc19jb21tYW5kLnB5) | `91.39% <50.00%> (-1.87%)` | :arrow_down: | | [pghoard/webserver.py](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618?src=pr&el=tree&filepath=pghoard%2Fwebserver.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open#diff-cGdob2FyZC93ZWJzZXJ2ZXIucHk=) | `89.42% <88.18%> (+0.01%)` | :arrow_up: | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/Aiven-Open/pghoard/pull/618/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Aiven-Open)