Forcing real-time update of RRD files while "on-demand" updating is active

[eschoeller]

I am working on setting up a multi-polling environment and realized I did not have the "Enable On-demand RRD Updating" option set on the master server. First, it would be nice for the remote poller to kick back an error in the log file reminding the user this needs to be configured ... otherwise you're lost.

Unfortunately I have some code that queries specific RRD files, does some calculations, and then returns a result back to Cacti as another data source item. The code ensures all the queried data is only about 90 seconds old, or it returns a NaN. Of course in an On-Demand environment this code will almost immediately fail.

Is there any desire to create some functionality within Cacti to flag certain data sources as needing real-time updates, and to skip the On-Demand process? Or, rather, what would be the best method for me to instruct Cacti to flush the output table for specific data source ids (in an effort to maintain the regular flush-data-to-disk every polling cycle methodology?) I've considered writing a dummy script which just asks for the graphs for each data source every 30 seconds, but I suspect that the overhead of that would be greater than simply leveraging something within the application itself to force an update of the RRD files.

[netniV]

From what I can see, there is not a direct way to force a full refresh of all your data sources. The boost_process_poller_output() function in lib/boost.php does take a local_data_id and if present attempts to find all entries matching that id.

However, the code that populates the poller_output_boost table isn't called at that point, it's merely read. So, if it's not there, it isn't processed.

Now, @cigamit may correct me on this because his understanding of this part is far better than mine.

[eschoeller]

It would be ideal I suppose if I could call graph_image.php directly from the command-line but that doesn't seem possible. I might try to hack another version of the script together to do that. I also saw cli/poller_output_empty.php but it doesn't appear to take any arguments (like a specific data source ID) but it might be a good start to take a more direct approach to the problem.

[netniV]

There is a graph_json.php (I think it's called). That effectively does the same job but returns as a json file. You still need to be authenticated but as long as you are, it will returned a graph which should in theory force retrieval of the data.

[cigamit]

Likely we enable the setting automatically when you have more than one data collector, and disable disabling it. How does that sound?

[eschoeller]

Well in the end I am planning to have multiple data collectors but I'll need to have certain data sources "flushed to disk" on every polling cycle. So I'm not sure how that mixes with your suggestion...

[cigamit]

Oh, I thought this was just settings confusion. Seems like you want a feature to mark certain data sources for immediate RRDtool update. Can you explain the user story a bit on this one?

[cigamit]

By the way, right now, you can simply call the function rrdtool_function_fetch($local_data_id) and that will flush the cache as well. However, that would only happen in say for example, the case of a plugin. Explain the story and then I can give it more thought.

[eschoeller]

Some of my data sources are used as inputs for other data sources. On each poller run the rrd files for these data sources are queried for the most recent data to then compute some new value and that output is used as the input for another data source.

As an example, to compute the total IT kW load for a data center I pull in the most recent kW readings for from all PDUs (via their RRD files) in that facility and then add them together. If those RRD files are only updated once every 30 minutes, my total IT kW load turns into a 30 minute average instead of a 1 minute average, which isn't terribly useful.

I could re-write all my code to use the 'rrdtool_function_fetch' function which would provide it a mechanism to get the data it needs whether it be in the boost output table or in an RRD file, but my code isn't necessarily "cacti aware". In some cases the RRD files themselves are simply hard-coded and the rrdtool command-line tools are used for all the work. I'll be the first to admit, not terribly ideal.

You've given me some good direction here though. It looks like I could actually call the 'boost_fetch_cache_check' or even the 'boost_process_poller_output' functions while passing a local_data_id to force Cacti to update the RRD files, and do this every 15-30 seconds to ensure I'm flushing the data promptly. Then my code would continue to work as it does today, perhaps with just a bit more latency in the "recency" of the data I'm using to make these computations. Then I could eventually make my code better;)

But if you see value in implementing a feature to flag these particular data sources as "immediate flush to disk" then I wouldn't need any work-around using either of these functions and a separate process to accomplish this task, also ensuring the rrd files I'm querying have the absolute most recent data that Cacti has collected. But I admit I'm probably a corner use case here.

[cigamit]

As long as you are using the rrdtool_function_fetch() call, the boost table will be processed for each local_data_id, if you are using another method, you will continue to have issues. Just use that call, and you will be all set.

[eschoeller]

I switched from using

boost_fetch_cache_check($id);

to this:

$time = time(true);
rrdtool_function_fetch($id,-1,$time,0);

The script is taking twice as long to run, but i'll see if there's a noticeable difference in the other behaviors.