Broken after upgrading to OpenNMS 1.8.4

[sabarirjpm]

Hi

I have been using this for quite sometime and this is indeed a great tool. We were able to fine tune our databases, look at the slave lag etc., Recently we upgraded to OpenNMS 1.8.4 and the data collection suddenly stopped. I am able to do an snmpwalk. I do not see it getting collected. I get the error, SnmpNodeCollector: genErr collecting data for node 172.16.30.27: Received genErr reqeusting oid .1.3.6.1.4.1.20267.200.1.1. Marking column is finished.

The SNMP version that is used on the host that runs mysql-agent is net-snmp 5.4.3. Can you please advice

[sabarirjpm]

Please find the debug output from collectd log

2010-10-18 12:55:20,176 WARN [DefaultUDPTransportMapping_127.0.0.1/0] SnmpNodeCollector: genErr collecting data for node 172.16.30.9: Received genErr reqeusting oid .1.3.6.1.4.1.20267.200.1.1. Marking column is finished. 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.2021.11.62 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.2021.11.63 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.2 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.3 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.4 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.5 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.6 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.7 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.8 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SingleInstanceTracker: Requesting oid following: .1.3.6.1.4.1.20267.200.1.9 2010-10-18 12:55:20,176 DEBUG [DefaultUDPTransportMapping_127.0.0.1/0] SnmpWalker: Sending tracker pdu of size 10

[seattlegaucho]

Apparently the issue appears when some of the values the script is trying to gather are undefined. For example, _ibufinserts, _ibufmerged and _ibufmerges (there might be others). In our test MySQL server there was no line matching:m/ merged recs, / as a result of SHOW ENGINE INNODB STATUS. This meant that the values corresponding to those entries weren't populated in the array confusing agentx.

The fix is to modify line 794 from: _$globalstatus{$key}{'value'} = $status->{$key};

to: _$globalstatus{$key}{'value'} = (defined($status->{$key}) and $status->{$key} ne '' ? $status->{$key} : 0);

This way, if a value is undefined, it gets populated with a "0" and agentx is happy.

I hope this helps and get put back into the main branch.

My $.02 G PS: I couldn't figure out a better hightlight/markup on this post.

[masterzen]

Thanks for the fix. That would be awesome if you could send me an output of SHOW INNODB STATUS for your version of MySQL, that way I'll add this to the test system to make sure we don't regress.

[seattlegaucho]

You may also want to change it to:

SHOW /!51000 ENGINE / INNODB STATUS

Since the old syntax is deprecated in 5.1 and removed in 5.5

Here's the output:

mysql -e "show engine innodb status\G" *** 1. row *** Type: InnoDB Name:

Status:

110203 14:22:11 INNODB MONITOR OUTPUT

Per second averages calculated from the last 4 seconds

BACKGROUND THREAD

srv_master_thread loops: 38 1_second, 38 sleeps, 2 10_second, 21 background, 21 flush

srv_master_thread log flush and writes: 38

SEMAPHORES

OS WAIT ARRAY INFO: reservation count 9, signal count 9 Mutex spin waits 30, rounds 757, OS waits 1 RW-shared spins 11, rounds 330, OS waits 8 RW-excl spins 0, rounds 0, OS waits 0

Spin rounds per wait: 25.23 mutex, 30.00 RW-shared, 0.00 RW-excl

TRANSACTIONS

Trx id counter B27 Purge done for trx's n:o < B26 undo n:o < 0 History list length 8 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started, process no 28851, OS thread id 1233471808 MySQL thread id 163515, query id 514448 localhost root

show engine innodb status

FILE I/O

I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 166 OS file reads, 159 OS file writes, 64 OS fsyncs

0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

INSERT BUFFER AND ADAPTIVE HASH INDEX

Ibuf: size 1, free list len 0, seg size 2, 0 merges merged operations: insert 0, delete mark 0, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 553229, node heap has 1 buffer(s)

0.00 hash searches/s, 0.00 non-hash searches/s

LOG

Log sequence number 1875252 Log flushed up to 1875252 Last checkpoint at 1875252 0 pending log writes, 0 pending chkp writes

36 log i/o's done, 0.00 log i/o's/second

BUFFER POOL AND MEMORY

Total memory allocated 274726912; in additional pool allocated 0 Dictionary memory allocated 42201 Buffer pool size 16383 Free buffers 16208 Database pages 174 Old database pages 0 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 0, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 152, created 36, written 105 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s LRU len: 174, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

ROW OPERATIONS

0 queries inside InnoDB, 0 queries in queue 1 read views open inside InnoDB Main thread process no. 28851, id 1201736000, state: waiting for server activity Number of rows inserted 5, updated 1, deleted 0, read 35

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s

END OF INNODB MONITOR OUTPUT

[seattlegaucho]

Oops! didn't expect the formatting to get so screwed, I hope it still works fine

[masterzen]

Yes, there's no issue I can get access to the raw output.

[masterzen]

I believe the latest v1.2 version fixes this issue.