search

jasonmcintosh / rabbitmq-zabbix

Zabbix RabbitMQ Configuration
Apache License 2.0
251 stars 168 forks source link

I don't know why fail to discovery queues. #47

Closed plutoid closed 8 years ago

plutoid commented 8 years ago

hi,

now the zbx server could get the data but RabbitMQ Refresh Rabbit Queue Status, it's so strange, could you give any suggestions?

seems the discovery rules didn't work.

root@v01:/etc/zabbix/scripts/rabbitmq# zabbix_sender -c /etc/zabbix/zabbix_agentd.conf -i /tmp/tmpnGsuad -vv zabbix_sender [26100]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 250; total: 250; seconds spent: 0.001954"}] info from server: "processed: 0; failed: 250; total: 250; seconds spent: 0.001954" zabbix_sender [26101]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 74; total: 74; seconds spent: 0.000586"}] info from server: "processed: 0; failed: 74; total: 74; seconds spent: 0.000586" sent: 324; skipped: 0; total: 324

Now the soft versions: Zabbix 3.0.3 RabbitMQ 3.2.4, Erlang R16B03 Linux v01 3.13.0-86-generic #130-Ubuntu SMP zabbix_sender has been installed. And I have activated UnsafeUserParameters=1 in zabbix_agentd.conf.

BTW: some tips for deploying this repo:

BRs, Hua

jasonmcintosh commented 8 years ago

IF as zabbix you actually run that command:

/etc/zabbix/scripts/rabbitmq/list_rabbit_queues.sh

That should return a JSON string of the queue names. I'd check with that. Then, see if on the zabbix server there are any issues with the auto discovery on the node. E.g. the template item for autodiscovery should tell you if autodiscovery fails.

plutoid commented 8 years ago

hi jasonmcintosh,

thanks for your reply very much!

this command does work, but I'm not sure where to check template item issue.

root@v01:/etc/zabbix/scripts/rabbitmq# /etc/zabbix/scripts/rabbitmq/list_rabbit_queues.sh
{"data": [{"{#VHOSTNAME}": "/", "{#QUEUENAME}": "async_create_task_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": "cbe_collect_bind_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": "cbe_collect_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": "cbe_content_raw_event_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": "cbe_content_tag_stat_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": "cbe_customer_event_queue"}, {"{#VHOSTNAME}": "/", "{#QUEUENAME}": 
...

No obvious helpful info found on zbx server log.


  3432:20160818:141513.003 __zbx_zbx_setproctitle() title:'trapper #5 [processed data in 0.002910 sec, waiting for connection]'
  3430:20160818:141513.004 __zbx_zbx_setproctitle() title:'trapper #3 [processing data]'
  3430:20160818:141513.004 trapper got '{"request":"sender data","data":[{"host":"v01","key":"rabbitmq.queues[/,queue_message_stat
s_deliver_get,smartlist_cbe_stat_queue]","value":"0"},{"host":"v01","key":"rabbitmq.queues[/,queue_message_stats_publish,smartlist
_cbe_stat_queue]","value":"487"},{"host":"v01","key":"rabbitmq.queues[/,queue_memory,smartlist_customer_event_queue]","value":"140
56"},{"host":"v01","key":"rabbitmq.queues[/,queue_messages,smartlis
......
ue":"20"},{"host":"v01","key":"rabbitmq.queues[/,queue_message_stats_deliver_get,wechat_fans_import_worker_queue]","value":"8"},{"host":"v01","key":"rabbitmq.queues[/,queue_message_stats_publish,wechat_fans_import_worker_queue]","value":"8"}]}'
  3430:20160818:141513.004 In recv_agenthistory()
  3430:20160818:141513.004 In process_hist_data()
  3430:20160818:141513.005 In process_mass_data()
  3430:20160818:141513.005 End of process_mass_data()
  3430:20160818:141513.005 End of process_hist_data():SUCCEED
  3430:20160818:141513.005 In zbx_send_response()
  3430:20160818:141513.005 zbx_send_response() '{"response":"success","info":"processed: 0; failed: 74; total: 74; seconds spent: 0.000730"}'
  3430:20160818:141513.005 End of zbx_send_response():SUCCEED
  3430:20160818:141513.005 End of recv_agenthistory()

BRs, Hua

plutoid commented 8 years ago

hi ,

I find this warning info- 'Global regular expression "rabbitmq vhosts for discovery" does not exist.' , I'm not sure if it will help debug. discoveryrules

jasonmcintosh commented 8 years ago

Ahh :) From readme file:

Low level discovery of queues, including GLOBAL REGULAR EXPRESSIONS:

https://www.zabbix.com/documentation/3.0/manual/regular_expressions The low level discovery, which is what determines what queues to be monitored, requires with the existing template that a filter be defined as a global regular expression.

I'm sorely tempted to delete the dependency on that, or if nothing else, need to update the docs to be more explicit.

Create a global regular expression (it's in the zabbix config area) @rabbitmq vhosts for discovery

You can just do "*" in there as I recall.

plutoid commented 8 years ago

hi,

thanks a lot!

there is one filter already, is that right? seems I don't have basic knowledge to understand your concepts, do you mean I need change "Label Marcro" value to "*"?

filter

update:

seems here should define global RE, which "EXPRESSION　ＴYPE" I should choose? globalre

update:

I try below RE, seems it works. rb_re rb_re_1

rb_re_2

jasonmcintosh commented 8 years ago

SO I'm assuming you're fixed now? There IS an outstanding bug where if you filter queues, the return code on queue refresh is not 0. I've not had time to work on that due to a different role at work where I'm no longer doing much RabbitMQ/Zabbix work.

somedrunkbum commented 8 years ago

@jasonmcintosh "" will not work, "." should be used if we want to discover all queues. Could you please add some simple example in documentation on that? That was a bit confusing for me. Thanks!

plutoid commented 8 years ago

@jasonmcintosh thanks for your reply, I'm keeping monitoring the alerts that have triggered, I'm not sure if that's working all right.

hamiltonfvi commented 8 years ago

@plutoid can you illustrate more about what you did to make it work? I had the same problems as you, I did the same modifications you did but it is still not working at all.

rabbitmq_zabbix.log still shows me "Found return code of 2" and nothing processed. I tried debug by adding -vv to zabbix_sender in the api.py script but the output doesnt say much in the logs.

jasonmcintosh commented 8 years ago

I merged a pull request to remove the global filter from the template. This should hopefully solve a lot of these issues! Let me know if this fixes the original issue please!

hamiltonfvi commented 8 years ago

I solved the issue by adding the Global Regular Expression ^.*queue$ and correcting some mistakes I had in the zabbix_agentd.conf For example, in hostname I used to have Zabbix hostname server which is incorrect. After I corrected the zabbix configuration file, placed the scripts files in the correct route including the Auth file, restarted the Zabbix agent, removed the template for the RabbitMQ host and added it back, finally, I got it working. 👍

I think you should review the documentation of the installation for make the installation easier for newbies like me, the way it is got me stuck for a long time, but at the end, it was worth it to have this working, your project is a huge life saver so receive a sincere thanks. 💯

I created a list of things to remember in case I need to recreated everything again, what I did was to put your documentation along with other's people notes and my own notes. The instructions are below in case you want to take some notes too. I hope it can be some help to others like me.

First and this is very important, the installation has to be done on the RabbitMQ Client Server, the one with RabbitMQ properly installed and working. Make sure also that zabbix_sender is installed. The template only works in Zabbix Server version 3.0 or higher you must have it before the installation**

Note: (I'm not sure if this step is needed, but it's working the way it is)

Make sure the information inside of this file is accurate with your environment, take some time to verify the route of Zabbix Configuration file and RabbitMQ Web Interface credentials.

USERNAME=guest PASSWORD=guest CONF=/etc/zabbix/zabbix_agentd.conf

Note: Remember to give Zabbix permissions to read the file.

Note: Make sure that Parameters like Server, ServerActive and Hostname are correct, this is very important for zabbix_sender and discovery to work. Remember that Hostname has to match with the one in Zabbix Server Web Interface.

UserParameter=rabbitmq.discovery_queue,/etc/zabbix/scripts/rabbitmq/list_rabbit_queues.sh UserParameter=rabbitmq.discovery_nodes,/etc/zabbix/scripts/rabbitmq/list_rabbit_nodes.sh UserParameter=rabbitmq[*],/etc/zabbix/scripts/rabbitmq/rabbitmq-status.sh $1 $2 $3

Note: Make sure the scripts are in the correct path.

What to do in case the installation fails?!

For start, you can check out the Zabbix logs generated by the API script.

    # tail -f /var/log/zabbix/rabbitmq_zabbix.log

You can change the level of information of the API script in order to get more relative information regarding zabbix_sender:

    # /etc/zabbix/scripts/rabbitmq.api.py

level=logging.INFO (partial information about what zabbix sender is sending) level=logging.WARNING (some information about what zabbix sender is sending) level=logging.DEBUG (detail information about what zabbix sender is sending)

So make the changes you need and keep and eye on the Zabbix logs mentioned before.

Additional Information. Queues to be Monitored.

You can also add a filter in this file to restrict which queues are monitored. This item is a JSON-encoded string. The format provides some flexibility for its use. You can either provide a single object or a list of objects to filter. The available keys are: status, node, name, consumers, vhost, durable, exclusive_consumer_tag, auto_delete, memory, policy

For example, the following filter could find all the durable queues: FILTER='{"durable": true}'

To only use the durable queues for a given vhost, the filter would be: FILTER='{"durable": true, "vhost": "mine"}'

To supply a list of queue names, the filter would be: FILTER='[{"name": "mytestqueuename"}, {"name": "queue2"}]'

jasonmcintosh commented 8 years ago

SO latest commit should help with a few issues - I ALSO now support setting a log level in the .rab.auth file (really it's become a config file). Please let me know if this fixes this so can close the issue!

hamiltonfvi commented 8 years ago

It does works good. Well done.

untitled1

hamiltonfvi commented 8 years ago

One note. Perhaps you should change type of information of the item RabbitMQ Refresh Rabbit Shovel Status to Character because it returns a value like "[]". Zabbix wont recognize it as numeric. I fixed mine after taken the screenshots and now is working fine.

untitled3

untitled

jasonmcintosh commented 8 years ago

@hamiltonfvi I'm going to close this open a new enhancement on the refresh shovel status. Ideally it'd return a return code vs. an array and behave similarly to the refresh queue status method.