benchmarks: some seem to be flushing layers forever

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.78k stars 430 forks source link

benchmarks: some seem to be flushing layers forever #8712

Closed koivunej closed 2 months ago

koivunej commented 2 months ago

Clearly issues related to layer flushes, or the benchmark not waiting long enough before starting.

test_download_churn[release-pg14-github-actions-selfhosted-10-std-fs-30]
test_layer_map[release-pg14-github-actions-selfhosted]

koivunej commented 2 months ago

This might actually be fallout from #8550. That made the wait_for_upload_queue_empty check fail.

We should really get rid of all wait_for_upload_queue_empty and instead have a checkpointing mode where we provide the lsn (for example, received from flush_ep_to_pageserver) and checkpoint waits until remote_consistent_lsn is at that (uploads have completed).