search

untergeek / zabbix-grab-bag

This is a collection of miscellaneous scripts for Zabbix data collection, maintenance, etc.
Other
106 stars 30 forks source link

Trapper Items still failing #13

Open mdiorio opened 8 years ago

mdiorio commented 8 years ago

So revisiting this months later as it's getting much more critical for us to monitor ElasticSearch now that we're getting ready to go to production with it and the trapper is still failing, with the same error. Was hoping this would be resolved after moving to Zabbix 3.2.0

2016-10-17 15:33:16,875 DEBUG zbxsender send_to_zabbix:58 Got response from Zabbix: {u'info': u'processed: 0; failed: 5; total: 5; seconds spent: 0.000015', u'response': u'success'}

1830:20161017:160013.892 __zbx_zbx_setproctitle() title:'trapper #1 [processing data]' 1830:20161017:160013.893 trapper got '{ "request":"sender data", "data":[ { "host":"la-1sesmaster01", "key":"health[initializing_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[relocating_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[unassigned_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[delayed_unassigned_shards]", "key":"health[initializing_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[relocating_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[unassigned_shards]", "value":0, "clock":1476734416.88}, { "host":"la-1sesmaster01", "key":"health[delayed_unassigned_shards]", "value":0, "clock":1476734416.88},", { "host":"la-1sesmaster01", "key":"nodestats[process.open_file_descriptors]", "value":271, "clock":1476734416.88}] }' 1830:20161017:160013.893 In recv_agenthistory() 1830:20161017:160013.893 In process_hist_data() 1830:20161017:160013.893 End of process_hist_data():SUCCEED 1830:20161017:160013.893 In zbx_send_response() 1830:20161017:160013.893 zbx_send_response() '{"response":"success","info":"processed: 0; failed: 5; total: 5; seconds spent: 0.000016"}' 1830:20161017:160013.893 End of zbx_send_response():SUCCEED

Any thoughts? Thanks!

untergeek commented 8 years ago

I'm terribly sorry for the delay. I've been so busy with regular work. This is only a side project for me, and I have been lax in its upkeep.

I'll try to get to it as soon as I can.

untergeek commented 8 years ago

I'm likely to switch libraries, too. Python Protobix looks promising, and is being actively developed.

Please stay tuned.

mdiorio commented 8 years ago

Looks like it might be a problem with the braces.

Using zabbix_sender, clusterstats[indicies.docs.count] fails. I created a new item with key indices.docs.count and sent a value to that and it worked.

untergeek commented 8 years ago

I don't see a problem on my end:

zabbix_sender -v -z 127.0.0.1 -s es_host -k clusterstats[indices.docs.count] -o 55665321
info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000084"
sent: 1; skipped: 0; total: 1
untergeek commented 8 years ago

I think it's time to switch to using Protobix, as the keys are correct, but something in the protocol seems to have shifted and zbx_send doesn't work any more.

untergeek commented 8 years ago

Protobix 1.0 isn't ready for release yet. In the meantime, I think that changing the items from trapper to zabbix_agent type will do the trick. That seems to work, where zbx_send does not.

mdiorio commented 8 years ago

Sure I can. I was. But pulling the metrics we need was driving up cpu to 50+% on the elastic nodes. I guess I'm going to have to scale back the metric collection frequency until batch works again.

superawesome commented 8 years ago

The problem is here: https://github.com/pistolero/zbxsend/issues/12

es_stats_zabbix calls zbxsend, which puts in a "clock" item that is formatted incorrectly. This causes Zabbix to reject the packet.

zabbix_sender works fine, because it doesn't use zbxsend.

It also works fine in Zabbix-2.4. Starting with 3.0 Zabbix actually validates this field, and hence it breaks. https://support.zabbix.com/browse/ZBX-10234