Failure in `test_hot_standby_gc`

[jcsp]

https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8591/10488910985/index.html#suites/950eff205b552e248417890b8b8f189e/c8bc7f8c49c04b80

psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 26922 failed: FATAL:  [NEON_SMGR] [shard 0] could not read block 0 in rel 1664/0/2676.0 from page server at lsn 0/016A6020
DETAIL:  page server returned error: Bad request: tried to request a page version that was garbage collected. requested at 0/16A6020 gc cutoff 0/45EAF70

This test is specifically exercising that the pageserver doesn't GC the data that a lagging replica needs, so this failure mode looks like a real bug.

[jcsp]

Looking at the code/test, it's not obvious what guarantees that the standby_horizon feedback should reach the pageserver before the GC is run:

• We need to see a SafekeeperTimelineInfo arrive at the pageserver between the standby postgres starting, and the test calling into GC.
• I note that the the safekeepers don't update standby_horizon from peers in record_safekeeper_info, although this test runs with just a single safekeeper anyway.

I can make this test fail reliably by running multiple safekeepers and stopping one of them partway through the test, so it seems likely that there is something unsound about how we're relying on standby_horizon updates to propagate, which might come up (rarely) even with a single safekeeper.