Spine/boost irregularity with scripts exceeding 512 character output limit

[eschoeller]

I have a script which has an output of greater than 512 characters. This is obviously a problem for several reasons, but there is a notable difference in how this situation is handled between spine and boost.

With spine, it does it's best to handle as much of the data as it can. I have a data source with 5 data source items, and it's usually able to pickup 3/5 or even 4/5 of the items, since these come at the tail end of the script output. The other items exceeding the 512 character limit get stored as NaN values.

Boost simply gives up. None of the data source items are stored. It throws these types of errors:

ERROR: /cacti/cacti-1.1.38/rra/xxx_rawsck_31.rrd: expected 4 data source readings (got 3) from 1525661656
ERROR: /cacti/cacti-1.1.38/rra/xxx_rawsck_31.rrd: found extra data on update argument: 0

This is unfortunate, but the root cause is really the script exceeding the result buffer of spine. What is also strange is the message "expected 4 data source readings (got 3)" which isn't correct since this data source has 5 items, as shown here (during a poller run without boost, and hacking the output to be under the 512 limit);

POLLER: Poller[1] Parsed MULTI output field 'totsck:909' [map totsck->totsck]
POLLER: Poller[1] Parsed MULTI output field 'tcpsck:36' [map tcpsck->tcpsck]
POLLER: Poller[1] Parsed MULTI output field 'udpsck:15' [map udpsck->udpsck]
POLLER: Poller[1] Parsed MULTI output field 'rawsck:0' [map rawsck->rawsck]
POLLER: Poller[1] Parsed MULTI output field 'ip_frag:0' [map ip_frag->ip_frag]
POLLER: Poller[1] CACTI2RRD: /usr/local/rrdtool-1.7.0/bin/rrdtool update /cacti/cacti-1.1.38/rra/xxx_rawsck_31.rrd --template totsck:tcpsck:udpsck:rawsck:ip_frag 1525662766:909:36:15:0:0

For reference, here is an example of the script output.

SCRIPT: perl /cacti/cacti-1.1.38/scripts/sarfetch.pl 'xxx.yy.com' '2' 'community' '161' '1500', output: CPU:all user:0.02 nice:0.00 system:0.03 iowait:0.00 steal:0.00 idle:99.95 procs:0.00 cswchs:744.00 pgpgins:0.00 pgpgouts:860.00 faults:51.00 majflts:0.00 pgfrees:96.00 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:326679964 kbmemused:70327844 memused:17.71 kbbuffers:823616 kbcached:48944300 kbswpused:110702960 swpused:23.85 dentunusd:437150 file_sz:4096 inode_sz:208275 pty_sz:16 runq_sz:0 plist_sz:1996 ldavg_1:1.82 ldavg_5:2.52 ldavg_15:2.72 totsck:764 tcpsck:34 udpsck:15 rawsck:0 ip_frag:0 tcp_tw:1953

If someone can point me in the direction of the buffer to increase in spine I would really appreciate it ... and maybe perhaps consider making this a command line argument that is adjustable within the UI :)

And my apologies for spamming so many issues the past couple days!

[netniV]

Issues are always welcome to make the platform more stable. The field in question is first defined within the configure.ac as results_buffer with a value of 1024. You could increase that but it should be noted that it would increase used everywhere so spine itself would take up more memory all around.

That value gets converted into RESULTS_BUFFER as a define statement.

[eschoeller]

Thanks I’ll recompile and see how that works

[cigamit]

We keep the width of the output column narrow in order to conserve space for memory tables. If using SSD or PCIe attached flash and the recommended MySQL settings (INNODB_FILE_PER_TABLE=ON), then it's possible to use InnoDB for the boost table and widen the column. The regular poller_output table you can widen as you see fit, and it's fine as a memory table.

[cigamit]

Close if you are cool with that.

[eschoeller]

So are you saying that this is more an issue with the width of the output column and not an issue with the RESULTS_BUFFER in spine?

[cigamit]

The results buffer is 1024, so you don't have to change spine, just increase the 'output' column width in the poller_output and poller_output_boost tables.

[eschoeller]

Would you consider adding a logging statement that indicates the poller_output and/or poller_output_boost table width has been overrun? I sort of had no idea this was happening. Currently 1.6% of my poller_output_boost table has 512 characters in the output column.

[eschoeller]

dang, looking over this output ... these are unsolved mysteries I've been battling for years. :(