Added support for RDS Clusters discovery / fixed item key ELBv2 namin…

[diegosainz]

Fixes issue #25

This PR adds support to discover RDS clusters and fixes an issue with ALB key names in the Zabbix template.

Changes to discovery

When discovery/rds.py has the additional argument clusters it will discover RDS clusters providing the following keys:

{#CLUSTER_ID}
{#CLUSTER_ENDPOINT}
{#CLUSTER_READER_ENDPOINT}

Changes to the Zabbix template

NOTE: tested under Zabbix 4.0

ALB discovery key naming errors

Zabbix 4 (maybe before?) throws the following error when importing the template:

Invalid key "instances[{#TARGET_GROUP_LOAD_BALANCER_NAME}][{#TARGET_GROUP_NAME}]" for item prototype "ELBv2: {#TARGET_GROUP_LOAD_BALANCER_NAME} target group {#TARGET_GROUP_NAME} total targets" on 
"Cloudwatch Template": incorrect syntax near "[{#TARGET_GROUP_NAME}]".

This is due to the fact that the key format with doulbe brackets is not supported (KEYNAME[][] - in fact I think it was an error before). So now it is changed to instances[{#TARGET_GROUP_LOAD_BALANCER_NAME}-{#TARGET_GROUP_NAME}] which will generate keys such as:

• instances[my_alb_01-targetGroup01]
• instances[my_alb_01-targetGroup02]
• instances[my_alb_02-targetGroup01]
• instances[my_alb_02-targetGroup01]

New item prototypes

Added a new discovery: "RDS cluster discovery", which creates the following item prototypes:

1. Cluster {#CLUSTER_ID} CPU usage
2. Cluster {#CLUSTER_ID} allocated storage

No other alerts/graphs prototypes were added since the individual instances will be already monitored by the RDS discovery.

[sherrerq]

Dear @diegosainz , Thanks for your help again! RDS clusters are well monitored now.

I am not sure why, but I can not import your new template. This error appears:

"Cannot find item "cloudwatch.metric[120,UnHealthyHostCount,AWS/ApplicationELB,Maximum,{$REGION},"LoadBalancer={#TARGET_GROUP_LOAD_BALANCER_ARN},TargetGroup={#TARGET_GROUP_ARN}",{$ACCOUNT}]" on "Cloudwatch Template" used in graph prototype "ELBv2 {#TARGET_GROUP_LOAD_BALANCER_NAME} {#TARGET_GROUP_NAME} UnHealthyHostCount" of discovery rule "ELBv2 Discovery" on "cloudwatch_zabbix_DS"." I check it and it seems to be correct for me. In any case, this is not a problem for me because I can already monitor this metric.

I have a new ask/challenge for you related a data obtained from S3. Do you know who to convert B into GB? I have changed the "Unit" value from B to G but it does not perform well.

[diegosainz]

Thanks for your help again! RDS clusters are well monitored now.

You're welcome. I'm glad you got it working.

I am not sure why, but I can not import your new template. This error appears: "Cannot find item "cloudwatch.metric[120,UnHealthyHostCount,AWS/ApplicationELB,Maximum,{$REGION},"LoadBalancer={#TARGET_GROUP_LOAD_BALANCER_ARN},TargetGroup={#TARGET_GROUP_ARN}",{$ACCOUNT}]" on "Cloudwatch Template" used in graph prototype "ELBv2 {#TARGET_GROUP_LOAD_BALANCER_NAME} {#TARGET_GROUP_NAME} UnHealthyHostCount" of discovery rule "ELBv2 Discovery" on "cloudwatch_zabbix_DS"."

mmmhhh.. I cannot replicate this error on Zabbix 4.0. What version of Zabbix are you using? It migh be the options you are using to import it. For example, a successful import for me was done with the following options:

Do you know who to convert B into GB? I have changed the "Unit" value from B to G but it does not perform well.

We really shouldn't use this medium for general issues. The Zabbix forums are a better place for this kind of question, but you should use the unit that the data is in. If the data is in bytes you should use B and then Zabbix would automatically convert it to KB, GB as required (check the item documentation under data type)

[sherrerq]

Dear @diegosainz , I have the same versión and I can not import it but do not worry, It is not important for me. You have helped me a lot with many clues. Di you have some blog or something similar where I can follow you? Maybe in the future, I can return your favour. Best regards.

[diegosainz]

Hi @sherrerq the template question is also to know if there is something I have to fix in the Pull Request, we don't want to merge something that is failing ;)

As for the public presence I don't have a blog or other account, I keep a really low public profile, but you can contact me via keybase at https://keybase.io/disago

@wawastein I can see #27 and #26 are related to the syntax error under Zabbix 4.0. Let me know if you want me to strip that part from this commit and rebase once one of those two PR are merged.

[sherrerq]

Hi @diegosainz You are very kind and professional.

This is the error that appears when I try to import your template: Cannot find item "cloudwatch.metric[120,UnHealthyHostCount,AWS/ApplicationELB,Maximum,{$REGION},"LoadBalancer={#TARGET_GROUP_LOAD_BALANCER_ARN},TargetGroup={#TARGET_GROUP_ARN}",{$ACCOUNT}]" on "Cloudwatch Template" used in graph prototype "ELBv2 {#TARGET_GROUP_LOAD_BALANCER_NAME} {#TARGET_GROUP_NAME} UnHealthyHostCount" of discovery rule "ELBv2 Discovery" on "Cloudwatch Template diego".

My Zabbix version is : zabbix_server (Zabbix) 4.0.12

These are my item prototypes related to RDS. It is a mixture of instances and clusters:

Now, I have another problem related to RDS items because cluster items are not created by prototypes. The most strange is that even the RDS application not appear.

If I check the metric from the command line it performs well: /usr/lib/zabbix# ./cloudwatch.metric --interval 14400 --metric VolumeReadIOPs --namespace AWS/RDS --statistic Average --region eu-west-1 --dimension DBClusterIdentifier=xxxxxxxxx --account xxxxxxxxxxxx 0.0222222222222

Is it possible because you created a new discovery rule "aws.discovery[rds,{$REGION},{$ACCOUNT}, cluster]" and I am monitoring with the previous one "aws.discovery[rds,{$REGION},{$ACCOUNT}]"?

Find attached my template. I have modified it with the new RDS cluster discovery and some item prototypes. Feel free to test it and maybe it performs well in your environment. templates.txt

[sherrerq]

Dear @diegosainz

I had some problems to monitor Aurora serverless clusters. The script only get data from "standard" clusters. I solved this make some little changes in your script. I suppose it can be made better but at least it works! XD

rds.txt

Moving to another topic, I would like to ask you if you know how to get AWS data using tags. I am going to open a new issue asking for help but maybe you can help me again. There are no many people working in this kind of monitoring. All my resources are tagged by one tag call Project. Then, what I want is a filter in Zabbix to get all metric resources from a project. Now, all is filtered by an application.

Thank you!

[diegosainz]

Thank you, @sherrerq. I've been out of the city in work meetings, but I'll check it in the following days and adjust accordingly. I don't have access to serverless Aurora instances but I imagine that by looking at your modifications I can make the changes.

I would like to ask you if you know how to get AWS data using tags. I am going to open a new issue asking for help but maybe you can help me again.

Yeah, let's go over that in another ticket to see if there is anything in which I can help

[sherrerq]

Dear @diegosainz ,

I have used an already opened issue. It seems not very reviewed but..... I appreciate if you can take a look when you have free time.... https://github.com/wawastein/zabbix-cloudwatch/issues/18#issuecomment-590309268

[diegosainz]

@sherrerq Oh I know what is going on. The template you provided has errors. Please use the template provided in this pull request. If you use it it will import correctly and discover the clusters as expected. Take into account that the discovery will run once per hour (as configured by default, anyway, once imported you can modify those values).

You may run a diff between your template and the one provided here and manually adjust it, but that's outside the scope of this project. I would recommend to just use the code provided in this pull request and modify the template once you've loaded it.

[ycatoireDS]

A big thank you to you @diegosainz, it helped me a lot for the integration on Zabbix V4 !