neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
14.48k stars 419 forks source link

tests: fix a flake in test_sharding_split_compaction #8136

Closed jcsp closed 3 months ago

jcsp commented 3 months ago

Problem

This test could occasionally trigger a "removing local file ... because it has unexpected length log" when using the compact-shard-ancestors-persistent failpoint is in use, which is unexpected because that failpoint stops the process when the remote metadata is in sync with local files.

It was because there are two shards on the same pageserver, and while the one being compacted explicitly stops at the failpoint, another shard was compacting in the background and failing at an unclean point. The test intends to disable background compaction, but was mistakenly revoking the value of compaction_period when it updated pitr_interval.

Example failure: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8123/9602976462/index.html#/testresult/7dd6165da7daef40

Summary of changes

Checklist before requesting a review

Checklist before merging

github-actions[bot] commented 3 months ago

2910 tests run: 2793 passed, 0 failed, 117 skipped (full report)


Code coverage* (full report)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
cb3d8b38e91028e1f46c3397b2086444c5c28c4c at 2024-06-23T20:01:09.688Z :recycle: