jcsp
commented
9 months ago Bug triage notes:
gc_horizonoriginated as a product requirement, to enable users to configure the PITR by size (therefore cost to the user) as well as/instead of configuring it by walltime.gc_feedbackboolean currently impacts our behavior here as well. Needs to be true for us to proactively generate image layers to enable GC to proceed. It is currently off by default.- Enabling
gc_feedbackis risky if PITR settings (size or time) are low, because it will cause us to incur costs from generating very many image layers - The behavior we want: after the PITR interval, the synthetic size goes down. We may choose whether to change how we clean up data, or change how we calculate synthetic size.
- Where we calculate
next_gc_cutoffwe take the min() LSN of horizon_cutoff and pitr_cutoff. If our user-facing story does not talk about the size-based (horizon) one, then we should exclude it from synthetic size calculation: this would have the result of making synthetic size go down after PITR interval passes. - If we only fix synthetic size calculation, it means that the user setting PITR interval to zero and then back to a large value won't work: when they set it back to the large vlaue their synthetic size will go back up. it still works if the user is willing to really wait for their actually PITR interval to expire (rather than hacking it down to zero).
- Should we have a "drop history"/"shrink-wrap" button? Imagining a UI PITR silder, the UX should be that when they slide the PITR interval down, they should be prompted to confirm, because this will drop data: so that would be an alternative to having an actual button.
User stories:
- Alice: Only care about cost, don't mind waiting for PITR interval to expire: they only require us to fix our sythetic size calculation to be accurate so that after waiting a week, they aren't still paying for data they dropped.
- Bob: Cares about cost, and not willing to wait a PITR interval for their cost to go down: they want to e.g. set PITR to zero and then set it back to their usual amount, with the expectation that this results in their cost going down. They would be satisfied by either really dropping data, or having a 'sticky' synthetic size calculation that only charges the user from the highest LSN their PTR interval has ever pointed to.
- Charles: Wants the data gone (maybe they accidentally wrote something confidential to the database). They don't care what the synthetic size is, they just want a way to say "drop history" and be sure the data is dropped.
This ticket tracks the existence of
gc_horizon, which is a strange setting: it controls how much wal we retain in byte terms, but this isn't a user-visible feature -- users only configure WAL retention in terms of time (pitr_interval).We should probably just remove
gc_horizon, after checking that its removal doesn't have any unintended side effects.Original ticket:
Users care about this because they don't want to be billed for data they've dropped: if they write 10GB, then drop the table, then wait a week (default pitr_interval), the data should get dropped.