safekeeper: use protobuf for sending compressed records to pageserver

[VladLazar]

Problem

https://github.com/neondatabase/neon/pull/9746 lifted decoding and interpretation of WAL to the safekeeper. This reduced the ingested amount on the pageservers by around 10x for a tenant with 8 shards, but doubled the ingested amount for single sharded tenants.

Also, https://github.com/neondatabase/neon/pull/9746 uses bincode which doesn't support schema evolution. Technically the schema can be evolved, but it's very cumbersome.

Summary of changes

This patch set addresses both problems by adding protobuf support for the interpreted wal records and adding compression support. Compressed protobuf reduced the ingested amount by 100x on the 32 shards test_sharded_ingest case (compared to non-interpreted proto). For the 1 shard case the reduction is 5x.

[github-actions[bot]]

5650 tests run: 5420 passed, 3 failed, 227 skipped (full report)

---

Failures on Postgres 17

• test_sharding_split_failures[failure16]: debug-x86-64
• test_sharding_split_failures[failure18]: debug-x86-64
• test_sharding_split_failures[failure19]: debug-x86-64

  # Run all failed tests locally:
  scripts/pytest -vv -n $(nproc) -k "test_sharding_split_failures[debug-pg17-failure16] or test_sharding_split_failures[debug-pg17-failure18] or test_sharding_split_failures[debug-pg17-failure19]"

  Flaky tests (8)

Postgres 17

• test_ondemand_download_pg_xact[None]: debug-x86-64

• test_pg_waldump: debug-x86-64

• test_sharding_split_failures[failure17]: debug-x86-64

• test_sharding_split_failures[failure15]: debug-x86-64

• test_sharding_split_smoke: debug-x86-64

• test_sharding_backpressure: debug-x86-64

• test_storage_controller_smoke[combination_ooonn]: debug-x86-64

  Postgres 15

• test_pull_timeline[True]: release-arm64

Code coverage* (full report)

• functions: 31.0% (7940 of 25592 functions)
• lines: 49.1% (63021 of 128385 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
8b13a09a5c233d98abd4a0d3e59157e7db16d6fd at 2024-11-21T10:57:02.228Z :recycle:}