Support for RDS Aurora Clusters

[sherrerq]

Dear Diego,

As you recommend me I have opened a new issue to talk about RDS Aurora cluster in a new thread. I have posted here our last commentaries. To sum up, You have given me a perfect solution to monitor only rds clusters but my need is to monitor both (instances and clusters). Then, I asked you again for help. What I need is to merge both scripts because aws_discovery.py only can run one rds.py. It does not recognize any other script called rds-xxxxx.py

Thank you very much again.

Another problem I have is that I will need to monitor RDS Aurora cluster. In this case, the dimension should change from DBInstanceIdentifier to DBClusterIdentifier. Then, I should change or create a new rds.py because the current rds.py only works with DBInstanceIdentifier

I have no idea of Python. Would you mind help me? What changes will do you do in rds.py? Is it enough to only change rds.py or I have to change more files?

This is my rds.py file: [root@salt-master discovery]# cat rds.py

!/usr/bin/python

from basic_discovery import BasicDiscoverer class Discoverer(BasicDiscoverer): def discovery(self, args): response = self.client.describe_db_instances() data = list() for instance in response["DBInstances"]: storage_bytes = int(instance["AllocatedStorage"]) pow(1024, 3) ldd = { "{#STORAGE}": storage_bytes, "{#RDS_ID}": instance["DBInstanceIdentifier"] } data.append(ldd) return data

I should add to this file this information but I do not have an idea of python: response = self.client.describe_db_clusters() data = list() for cluster in response["DBClusters"]: storage_bytes = int(cluster["AllocatedStorage"]) * pow(1024, 3) ldd = { "{#STORAGE}": storage_bytes, "{#RDS_CLUSTER_ID}": cluster["DBClusterIdentifier"] } data.append(ldd) return data .......................................................................... To monitor Aurora RDS clusters you are in the right path. You don't really need to know Python to modify the script (just the basic syntax details). The current rds.py script will correctly discover the instances, but if you want to have also the data at cluster level the following should work:

!/usr/bin/python

from basic_discovery import BasicDiscoverer

class Discoverer(BasicDiscoverer): def discovery(self, *args): response = self.client.describe_db_clusters() data = list()

    for cluster in response["DBClusters"]:

        storage_bytes = int(cluster["AllocatedStorage"]) * pow(1024, 3)
        for instance in cluster["DBClusterMembers"]:

            data.append({
                "{#CLUSTER_ID}": cluster["DBClusterIdentifier"],
                "{#STORAGE}": storage_bytes,
                "{#MEMBER_ID}": instance["DBInstanceIdentifier"]
             })

    return data

This will output the following:

{ "data": [ { "{#CLUSTER_ID}": "myCluster01", "{#MEMBER_ID}": "myInstance01", "{#STORAGE}": 1073741824 }, { "{#CLUSTER_ID}": "myCluster01", "{#MEMBER_ID}": "myInstance02", "{#STORAGE}": 1073741824 } ] } ........................... [root@salt-master zabbix]# ./aws_discovery.py --service rds-bak --region eu-west-1 --account xxxxxx Traceback (most recent call last): File "./aws_discovery.py", line 43, in args.service, args.region) File "/usr/lib/zabbix/discovery/aws_client.py", line 15, in init region_name=region) File "/usr/lib/python2.7/site-packages/boto3/init.py", line 79, in client return _get_default_session().client(*args, *kwargs) File "/usr/lib/python2.7/site-packages/boto3/session.py", line 192, in client aws_session_token=aws_session_token, config=config) File "/usr/lib/python2.7/site-packages/botocore/session.py", line 835, in create_client client_config=config, api_version=api_version) File "/usr/lib/python2.7/site-packages/botocore/client.py", line 76, in create_client service_model = self._load_service_model(service_name, api_version) File "/usr/lib/python2.7/site-packages/botocore/client.py", line 114, in _load_service_model api_version=api_version) File "/usr/lib/python2.7/site-packages/botocore/loaders.py", line 132, in _wrapper data = func(self, args, **kwargs) File "/usr/lib/python2.7/site-packages/botocore/loaders.py", line 378, in load_service_model known_service_names=', '.join(sorted(known_services))) botocore.exceptions.UnknownServiceError: Unknown service: 'rds-bak'. Valid service names are: accessanalyzer, acm, acm-pca, alexaforbusiness, amplify, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, application-autoscaling, application-insights, appmesh, appstream, appsync, athena, autoscaling, autoscaling-plans, backup, batch, budgets, ce, chime, cloud9, clouddirectory, cloudformation, cloudfront, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codebuild, codecommit, codedeploy, codeguru-reviewer, codeguruprofiler, codepipeline, codestar, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, config, connect, connectparticipant, cur, dataexchange, datapipeline, datasync, dax, detective, devicefarm, directconnect, discovery, dlm, dms, docdb, ds, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecs, efs, eks, elastic-inference, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, es, events, firehose, fms, forecast, forecastquery, frauddetector, fsx, gamelift, glacier, globalaccelerator, glue, greengrass, groundstation, guardduty, health, iam, imagebuilder, importexport, inspector, iot, iot-data, iot-jobs-data, iot1click-devices, iot1click-projects, iotanalytics, iotevents, iotevents-data, iotsecuretunneling, iotthingsgraph, kafka, kendra, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, lex-models, lex-runtime, license-manager, lightsail, logs, machinelearning, macie, managedblockchain, marketplace-catalog, marketplace-entitlement, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediastore, mediastore-data, mediatailor, meteringmarketplace, mgh, migrationhub-config, mobile, mq, mturk, neptune, networkmanager, opsworks, opsworkscm, organizations, outposts, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, polly, pricing, qldb, qldb-session, quicksight, ram, rds, rds-data, redshift, rekognition, resource-groups, resourcegroupstaggingapi, robomaker, route53, route53domains, route53resolver, s3, s3control, sagemaker, sagemaker-a2i-runtime, sagemaker-runtime, savingsplans, schemas, sdb, secretsmanager, securityhub, serverlessrepo, service-quotas, servicecatalog, servicediscovery, ses, sesv2, shield, signer, sms, sms-voice, snowball, sns, sqs, ssm, sso, sso-oidc, stepfunctions, storagegateway, sts, support, swf, textract, transcribe, transfer, translate, waf, waf-regional, wafv2, workdocs, worklink, workmail, workmailmessageflow, workspaces, xray

Best Regards

[sherrerq]

Now, I can detect my Aurora cluster

[root@salt-master zabbix]# ./aws_discovery.py --service rds --region eu-west-1 --account xxxxxxxx {"data": [{"{#RDS_ID}": "instance-1", "{#STORAGE}": 1073741824}, {"{#CLUSTER_ID}": "cluster", "{#MEMBER_ID}": "instance-1", "{#STORAGE}": 1073741824}]}

Also, I can get data from the AWS metrics. ./cloudwatch.metric --interval 1200 --metric VolumeWriteIOPs --namespace AWS/RDS --statistic Average --region eu-west-1 --dimension DBClusterIdentifier=cluster --account xxxxxxxx 1193.33333333

Al seems to work fine but these items do not appear in my Zabbix dashboard. I mean, I have these metrics configured as an item prototype and it does not generate the items in my host ¿?¿? These are some of my item prototypes

The problem is that the cluster items are not created in my host and when I go to Monitoring/latest data only appear metrics from db instances, db clusters no appear.

¿Any ideas?

[diegosainz]

@sherrerq please check out the mentioned PR and test it on your side. From my tests it should work as expected. The Zabbix template is included. Take care if you are overwriting your current template to avoid losing any customizations you may have.