onemoredata / bagger

Massive log storage in PostgreSQL
BSD 2-Clause "Simplified" License
9 stars 0 forks source link

Dimensions versioning #48

Closed einhverfr closed 8 months ago

einhverfr commented 8 months ago

Partially addresses #6 Fixes #35 Fixes #41

 This PR adds temporal bounds for partitioning dimensions.  These bounds
 address the fact that we may need to change these and have them take
 effect consistently in the cluster some fixed number of hours out.  The
 future period affects both expirations and new dimensions.

 The logic is that if production is set in the config table to 0, we can
 safely backdate changes.  If production is set to 1, we cannot.  We
 then look for the next boundary for partitioning at least one hour out
 (and currently this assumes hourly partitioning, this could change
 after 1.0).  If dimensions_hrs_in_future is set we add that many hours
 instead.