--template support for rrdtool update --daemon

[oetiker]

It seems rrdupdate does not support --template in combination with --daemon. This functionality is needed to get Cacti working with rrdcached in the way rrdcached seems to be intended.

Please see [source:/trunk/program/src/rrd_update.c@2179#L438] and [http://bugs.cacti.net/view.php?id=1590 bugs.cacti.net].

[oetiker]

[oetiker] yes, at the moment rrd_daemon does not support the template option ... as the purpose of the template option is to check the structure of the rrd file on update, rrd_daemon would have to read the rrdfile header on update which would sort of defeat the purpose of it all (being fast and such) ...

the way out, is to move the whole updating mechanism into the daemon and sync the rrd file rather than just queuing the individual instructions ... this would open up a bunch of other neat optimization, like graphing without first dumping the file to disk ...

unfortunately this is a major coding effort and no one seems to be willing to invest into this.

[dacox]

Tobias, this is functionality I would love to see. I've only just started perusing the code today so I don't think I'll be submitting patches any time soon, but could you elaborate a little on what you think would be involved?

It sounds like you are suggesting building the rrd's in memory and then somehow merging them when a flush occurs.

[oetiker]

yes, the idea is to change the way the cache daemon works ... instead of caching the updates, it would defer writing to disk, but do all the math already ... but since this is major work, you may want to look at github.com/oetiker/rrdtool-2.x for talk about more fundamental changes ...

[msperl]

As a stop-gap, maybe it would be possible to implement reading of the rrd-file-structure once (cache it to avoid the repeated reads of the file) and then "expand" the passed update arguments if there is a "--template" argument. It is not optimal, but at least it would work and might make it usable for cacti or xymon.

[oetiker]

or you could be optimistic und cache the template along with the update instructions ... as long as there is no error all will be well ... I guess in many instances people could accept this limitation ...

[msperl]

OK - got an initial (not fully tested) version - see: https://github.com/msperl/rrdtool-1.x/commit/41a2cdae7904f99e59d7d9f7fa209e16a9793f58

Essentially it takes the template and expands it - not in the most efficient way i guess, but at least it works...

it also does a bit of caching on the "template" it extracts from the rrdfile - currently in a single linked list, which may not be sufficient and result in long list walks if more entries are used. Maybe we should switch to GTree like rrd_daemon does...

It also includes the option to strip "absolute" paths to relative ones by setting the environment variable RRDCACHED_STRIPPATH.

Xymon for example uses absolute paths and this allows to work around this by just defining this environment variable...

Please have a look - if that looks fine, then I will do some cleanups around caching, create more documentation with regards to RRDCACHED_STRIPPATH and create a pull request against 1.4.

[msperl]

OK - updated to use gtree for the caching. See https://github.com/msperl/rrdtool-1.x/compare/oetiker:1.4...1.4-cached%2Btemplate please review/comment

[oetiker]

two things

• since this is a new feature I really need to patch to be relative to MASTER
• the LIBS="${LIBS} -lglib-2.0" change seems odd ... code im acinclude.m4 should take care of this already.

[msperl]

• relative to master: see my "painpoints" about rrdtool with RH5
• For some reason CORE_LIBS needs it to compile, and as CORE_LIBS includes "LIBS" I added it to LIBS directly

Anyway: there is still an issues I found with compatiblity testing of xymon (memory related) - the "RRDCACHED_STRIPPATH" argument works without issues

[oetiker]

you did get my reply re rh5, right ? the dependencies between 1.4 and master are not changed ... or did I miss your point ?

look at the definition of EX_CHECK_ALL in acinclude.m4 this should happen already LIBS=${LIBS}" "$PKGCONFIG --libs-only-l $4

[msperl]

Well - there was a change between master and 1.4 that made it impossible to compile it on RH5 (even after creating .configure et.al. on a RH6 and copying them to the RH5 target system) - can not remember details now (dependencies on a specific cairo pango or something...)

As for EX_CHECK_ALL: you are right: changing the order (moving the glibc test up prior to CORE_LIBS="${LIBS}") alone was sufficient...

The latest version in my 1.4 tree has all those fixes (especially those stupid memory allocation fixes)

I will try to rebase it for you...

[msperl]

Rebase did not work for me - so patched manually and sent a pull request https://github.com/oetiker/rrdtool-1.x/pull/462