feat: scheduler detects peer survivability and clears offline peers' metadata

[BruceAko]

Description

1. The Host GC method will remove hosts that did not announce in time and their peers' metadata.
2. The scheduler Host GC interval is lowered in order to detect non-surviving hosts in time.
3. Add unit tests.

Related Issue

Resolve #3350

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Documentation Update (if none of the other choices apply)

Checklist

• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have added tests to cover my changes.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 52.67%. Comparing base (4f1046c) to head (12f02e8). Report is 5 commits behind head on main.

Additional details and impacted files

[](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) ```diff @@ Coverage Diff @@ ## main #3353 +/- ## ========================================== + Coverage 52.63% 52.67% +0.03% ========================================== Files 192 192 Lines 20428 20408 -20 ========================================== - Hits 10753 10750 -3 + Misses 8879 8855 -24 - Partials 796 803 +7 ``` | [Flag](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | `52.67% <100.00%> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | | |---|---|---| | [scheduler/resource/host.go](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353?src=pr&el=tree&filepath=scheduler%2Fresource%2Fhost.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-c2NoZWR1bGVyL3Jlc291cmNlL2hvc3QuZ28=) | `93.25% <100.00%> (+0.23%)` | :arrow_up: | | [scheduler/resource/host\_manager.go](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353?src=pr&el=tree&filepath=scheduler%2Fresource%2Fhost_manager.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-c2NoZWR1bGVyL3Jlc291cmNlL2hvc3RfbWFuYWdlci5nbw==) | `86.20% <100.00%> (+1.59%)` | :arrow_up: | | [scheduler/service/service\_v2.go](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353?src=pr&el=tree&filepath=scheduler%2Fservice%2Fservice_v2.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-c2NoZWR1bGVyL3NlcnZpY2Uvc2VydmljZV92Mi5nbw==) | `82.57% <100.00%> (+0.07%)` | :arrow_up: | ... and [9 files with indirect coverage changes](https://app.codecov.io/gh/dragonflyoss/Dragonfly2/pull/3353/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)