fixed size extents defined upfront; reasonable default 16KB?
fixed size will permit the re-use of extents if one stops being used.
it's easier
RRA (round-robin) notion modified to 'time' not rows
RRA1:
90 days of 5 minute CF <- user provides
25,920 rows <- rrd requires
13 16KB extents <- what rrd consumes
same thing, but less complicated -- users don't need complication
supports RRA0 idea much simpler
Extent offset tracking would likely need a larger header and/or extent based header
like an inode tracks to other inodes
use posix_fallocate() to avoid excessive IO
RRA0 stores 'literal storage': no consolidation, no normalization
RRA1+ would be CF (not support RAW ... anyone using RAW must use extent-growth on RRA0 to retain data)
RRAs would age by time, not by rows, given example:
RRA0 - store data for 3 days (settable) no matter how many datapoints inserted, sub-second supported (micro or nano second insanity even.)
RRA1 - a 60-step average store for 1 month
RRA2 - a 300-step average store for 1 year
RRAx - whatever
special RRA CF types like COMPUTE, Holt-Winters types, etc would only be able to operate on RRA1 not RRA0
RRA0 holding all data for RRA1+ would be able to coalesce writes
boundary step timing of updates into RRA0 would trigger a normalize/coalesce for a DS in other RRAs
then normalization is the same thing we have now with RRA1 an update passing it's boundary would normalize to RRA2, etc.
everyone cares about precision for 'what is happening now' RRA0 gives them exactly that
no one really cares about that precision a long time ago; RRA1+ gives them that.
btw: I'd use RRA1+ as 1-min/5-min and start collecting data more often in RRA0: 30,15,10,1 second intervals.
RAW DST - new datasource type
store exactly what was given at time it was given; no normalization
it's a minor issue to know up front all numeric types
people get confused about this, getting it wrong stores wrong information
getting it wrong is easy, example: riak/memcache/mysql all have stats output which give key/value and NO numeric type ... one must basically upfront guess at the type.
if rrd had a data type RAW then a system which was capable could pass in the DST dynamically and rrdtool would do the operation on export.
DST against RAW would require heartbeat:min:max to be passed in as well.
RAW could then be used for any numeric which was unknown at the cost of performance due to ignorance.
RAW could have an rrdtool modify feature which would convert RAW into another datatype when it was learned (for those so inclined.)
RAW would only be supported in RRA0
file-size currently is efficient 8bytes per datapoint; storing time in RRA0 will impact this
LZO compression built-in would be highly beneficial if definable per RRA:
super long-term datafiles
cold-storage rrds that won't be updated again.
could be set on RRA1+ (being CFs of RRA0 are not modified as often)
delayed RRA1+ CF ( consolidations )
tunable delayed compactions: as RRA1+ aren't required near-time they can be delayed
delay wouldn't work for algorithms like holt-winters and COMPUTE which would have to operate on RRA1 (not [0] due to above RRA0 implementation)
would reduce IO
delay randomized(offset) to avoid many consolidations in unison to benefit systems with multi-millions of rrd datafiles
DEF to support datafile list in RRD xport/graph syntax so offline cold-storage (compressed) rrds can be supported
note: two calls to rrd_update both inserting into the same time but for different ds-names each
note: think about a database with millions of datasources and the rrd-datafiles all open writable once-only (mmap()'ed) and the db constantly only updating with msync() no close()
note: OS overhead on open() is high enough to engineer around it ... only way to do that is store more in one file and mmap() that file, but we can't make a user 'coelesce' the data for writing like rrdtool requires now with the 'write once into time-frame model'
note: msync() will flush blocks to user/shared space and permit any external app to read it ( without having to go through the write database for reads )
fixed extent size during datafile creation
ensure first extent offsets into extent like rrd does now:
this actually still helps quite a lot even in a new format because IO creation won't all happen at the same 'moment' but be spread out over time.
only required on first extent, subsequent extents need not do this ( beneft is gained from the original offset ) and once max-extents is reached the first extent will be round-robin'ed to.
additions from ryan 2013-09-25