Reduce layer map locking on the read path

[VladLazar]

Currently, the read path (Timeline::get and Timeline::get_vectored when it merges) hold the layer manager read lock while accumulating the reconstruct data (includes on-demand downloads and disk IO). The non-vectored read path grabs the lock here.

Locking for this long is not required. Instead, the read path could hold on to a storage_layer::Layer (introduced in https://github.com/neondatabase/neon/pull/4938) and only hold the lock while querying the layer map. The fix for the vectored read path will be similar, but slightly more involved. The search fringe must be updated to hold storage_layer::Layer instead of layer descriptions and the in-memory handles will need some thought as well.

[koivunej]

Just realized while looking at some logs (slack thread), that compaction now needs to wait for on-demand downloads. This is a regression introduced with the struct Layer work. Before that, we released the read lock for layer downloading and switching.

Since no one has noticed this in quite some time, I don't think this is critical, but we should get it eventually fixed, so a slight boost to this issue.

What needs to be done:

• always drop the read lock before doing anything with the Layer

[VladLazar]

Update:

• Merged PR for the normal read path
• PR for vectored read is ready to merge, but I'm holding it back for the next release

[VladLazar]

Both PRs merged. Releasing on the week of 2024-04-08