neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
14.58k stars 423 forks source link

quantification of compaction algorithms #7770

Closed problame closed 1 month ago

problame commented 4 months ago

Child of 2024Q2 compaction work: https://github.com/neondatabase/neon/issues/8001

This epic tracks the efforts to quantify any compaction algorithm's outcomes.

We had a brainstorming session some time back to come up with an (incomplete) set of potentially useful metrics: https://www.notion.so/neondatabase/Productionize-Tiered-Compaction-eca9b06aa1ae4c62bdf6cf40ab002eb6?pvs=4

Meeting notes / ideas:

Demo test case to adapt / apply the Python helpers to:

### Refs
- [ ] https://github.com/neondatabase/neon/pull/7867
- [ ] develop automated workflow to back-test & perf-evaluate with production data
problame commented 4 months ago

This week:

problame commented 3 months ago

This week, @problame to address his follow-up requests from https://github.com/neondatabase/neon/pull/7867#pullrequestreview-2096797786

problame commented 1 month ago

This issue was part of

In the end, that work expanded into Q3 and we focussed solely on bottommost compaction.

Bottommost compaction is very deterministic and hence, the existing quantification work in test_gc_feedback (https://github.com/neondatabase/neon/pull/7867) is sufficient to qualify & quantify it.