test: compatibility test doesn't cover L1s, image layers, compaction, sharding, aux files

[jcsp]

When generating compat snapshots, we should ensure they include a reasonable cross section of how we use storage (currently they just contain a few L0s):

• L1 deltas
• image layers
• multiple shards
• AUX files (i.e. configure some logical replication stuff before writing the snapshot)

When loading such snapshots in forward/backward compat tests, we should do more checks than just that postgres can read:

• Check that compaction can still run

[jcsp]

CC @bayandin @kelvich

[koivunej]

After our bug triage call I looked around the test and test report, writing down some ideas here:

The test report for https://github.com/neondatabase/neon/commit/5c88213eaf1b1e29c610a078d0b380f69ed49a7e: https://neon-github-public-dev.s3.amazonaws.com/reports/main/6561781567/index.html#suites/0fbaeb107ef328d03993d44a1fb15690/9abdf4df085f5378/

notes on the test report

*Assuming* that the pageserver.log is the `compatibility_snapshot/repo/pageserver_1/pageserver.log` no compaction happens. Looking at the test code, only pgdump is ran before deleting the timeline. The timeline deletion is a bit suspicious, because it involves deleting the `local_fs_remote_storage`, something which I had not spotted before: https://github.com/neondatabase/neon/blob/6129077d31266361a79bf91cc9f95572e8fd39b5/test_runner/regress/test_compatibility.py#L461 Perhaps the removal of `local_fs_remote_storage` should be just removed.

ideas on how to trigger image layer creation and L0 => L1 compaction and gc when testing the snapshot

To trigger compaction and gc: we are currently producing a non-insignificant amount of data with 60s pgbench in `test_create_snapshot`. The defaults are used, and we end with a `timeline_checkpoint`, so there's a compaction before shutdown. This is different from what we see in staging and prod in the sense that with there's enough tenants to shutdown in all states, or only in the state that safekeepers have more WAL for us. That is good for having at least some compactions right at the startup, but I am unsure how to replicate this in our tests. If much less data was produced, the initial snapshot creation could run with the only one manual checkpoint (which includes compaction, but no gc). If we were then to add more data (run pgbench again) and do an immediate shutdown of pageserver, and a clean shutdown of safekeepers, perhaps then we would have conditions for during the snapshot testing phase: - pageserver always receiving more WAL during startup - compaction always after receiving the WAL We would have to wait for the lsn the second pgbench ended with for pageserver first, then do checkpoint (which includes compaction), and a gc. Getting compaction to always produce layers AND do L0 => L1 when testing the snapshot requires lowering the related configuration values, `image_creation_threshold` and for L0 => L1, making the `compaction_threshold` suitably small as well (to cover all of the L0s created by the second pgbench). Getting gc to do something always when testing the snapshot and before doing the sqldump this case is not easy either; we will need to configure the `pitr` and invoke it with a suitable `gc_horizon`. Might be that we need to first have a `CompactionResult` we can assert over at python land that the things we expect are actually happening. There might be one test checking the `GcResult`.