puppetlabs / puppetdb

Centralized Puppet Storage
http://docs.puppetlabs.com/puppetdb
Apache License 2.0
298 stars 224 forks source link

Consider regularly resetting factset stable/volatile balances #3956

Open rbrw opened 4 months ago

rbrw commented 4 months ago

In an attempt to decrease the steady-state Postgres write load PuppetDB currently stores each factset in two separate jsonb fragments, stable and volatile. Whenever a fact changes, the top-level subtree that contains it is moved from stable to volatile, if it wasn't already in volatile, and never moves it back. So if a leaf fact in a large subtree changes once, and then never changes again, the entire subtree containing that fact will always add to the write load when updating that factset, something the split is intended to avoid. (See the footnote in #3955 for additional information.)

To avoid that, and as a fairly simple improvement, consider moving all of a factset's volatile facts back into stable on some cadence. For example, perhaps make the move during any given factset update based on a percentage that would reset a fact every day/week/... assuming the default factset submission rate,

Thanks to Austin Blatt for the initial suggestion.

github-actions[bot] commented 4 months ago

Migrated issue to PDB-5742