Elastic Agent : UI is Missing about 104 Metrics Namespaces for the AWS Module

[joshuaquek]

Describe the enhancement: At the moment, it seems that we can only configure the following AWS Namespaces for metrics collection in our Elastic AWS Module ( available as manifest.yml files here - https://github.com/elastic/beats/tree/v7.17.0/x-pack/metricbeat/module/aws in each individual sub-folder):

• AWS/RDS
• AWS/DynamoDB
• AWS/EC2
• AWS/ELB
• AWS/ApplicationELB
• AWS/NetworkELB
• AWS/Kinesis
• AWS/Lambda
• AWS/NATGateway
• AWS/S3
• AWS/SNS
• AWS/SQS
• AWS/TransitGateway
• AWS/Usage
• AWS/VPN
• AWS/AmplifyHosting

This can be set in the AWS Module UI

However, the following AWS Namespaces are missing:

• AWS/ApiGateway
• AWS/AppRunner
• AWS/AppStream
• AWS/AppSync
• AWS/Athena
• AWS/Backup
• AWS/CertificateManager
• AWS/ACMPrivateCA
• AWS/Chatbot
• AWS/ClientVPN
• AWS/CloudFront
• AWS/CloudHSM
• AWS/CloudSearch
• CloudWatchSynthetics
• AWS/Logs
• AWS/CodeBuild
• AWS/CodeGuruProfiler
• AWS/Cognito
• AWS/Connect
• AWS/DataLifecycleManager
• AWS/DataSync
• AWS/DMS
• AWS/DX
• AWS/DocDB
• AWS/DAX
• AWS/ElasticGPUs
• AWS/EC2Spot
• AWS/AutoScaling
• AWS/ElasticBeanstalk
• AWS/EBS
• AWS/ECS
• AWS/ECS/ManagedScaling
• AWS/EFS
• AWS/ElasticInference
• AWS/GatewayELB
• AWS/ElasticTranscoder
• AWS/ElastiCache
• AWS/ElastiCache
• AWS/ES
• AWS/ElasticMapReduce
• AWS/MediaConnect
• AWS/MediaConvert
• AWS/MediaLive
• AWS/MediaPackage
• AWS/MediaStore
• AWS/MediaTailor
• AWS/Events
• AWS/FSx
• AWS/FSx
• AWS/GameLift
• AWS/GlobalAccelerator
• Glue
• AWS/GroundStation
• AWS/HealthLake
• AWS/Inspector
• AWS/IVS
• AWS/IoT
• AWS/IoTAnalytics
• AWS/IoTSiteWise
• AWS/ThingsGraph
• AWS/KMS
• AWS/Cassandra
• AWS/KinesisAnalytics
• AWS/Firehose
• AWS/KinesisVideo
• AWS/Lex
• AWS/Location
• AWS/LookoutMetrics
• AWS/ML
• AWS/Kafka
• AWS/AmazonMQ
• AWS/Neptune
• AWS/NetworkFirewall
• AWS/NetworkManager
• AWS/OpsWorks
• AWS/Polly
• AWS/PrivateLinkEndpoints
• AWS/PrivateLinkServices
• AWS/QLDB
• AWS/Redshift
• AWS/Robomaker
• AWS/Route53
• AWS/Route53RecoveryReadiness
• AWS/SageMaker
• AWS/SageMaker/ModelBuildingPipeline
• AWS/SDKMetrics
• AWS/ServiceCatalog
• AWS/DDoSProtection
• AWS/SES
• AWS/S3/Storage-Lens
• AWS/SWF
• AWS/States
• AWS/StorageGateway
• AWS/SSM-RunCommand
• AWS/Textract
• AWS/Timestream
• AWS/Transfer
• AWS/Translate
• AWS/TrustedAdvisor
• AWS/IPAM
• AWS/WAFV2 for AWS WAF resourcesWAF for AWS WAF classic resources
• AWS/WorkMail
• AWS/WorkSpaces
• AWS/WorkSpacesWeb

Would there be any roadmap to support these in the UI?

As of now the workaround solution is to manually build the YAML from https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html and input it inside the Cloudwatch metric dropdown:

The issue with this is that one cannot set differing intervals for each manually set namespace & metric entered into this section, but uses a single interval specified under this dropdown.

Describe a specific use case for the enhancement or feature:

To be able to define different intervals for different metric namespaces, like how it is already possible for DynamoDB and EC2 for example.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kaiyan-sheng]

cc @ravikesarwani

[ravikesarwani]

Correct, we have certain number of most common out of the box integrations available and based on customer demand will continue to add more over time. As you mentioned the generic AWS CloudWatch integration can be used for other metricset we don't have built in out of the box integrations available right now.

Regarding the issue around differing collection interval, you will need to add multiple integrations with common set of metricest grouped where the same collection internal would suffice the requirements.

[joshuaquek]

Ok got it! Thanks @ravikesarwani

Ok so it seems that now the workaround to this challenge is to set different intervals for different missing namespaces, and then add multiple "AWS Module" integrations to a single policy. After that in each "AWS Module" integration attached to the policy, tick only the Cloudwatch metrics and input the custom YAML for that missing namespace and set the interval.

Would this be correct?

[joshuaquek]

Also, for AWS/ElasticCache namespace, there are two different types of resources, Redis and Memcached.

How can I construct the YAML for this? Since the metrics available for Redis and Memcached are different:

• Redis - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html
• Memcached - https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/CacheMetrics.Memcached.html

Could we possibly use the resource_type YAML field for this (I am unable to find what the exact string is, if we can use this field)? Since they belong under the same AWS Namespace of AWS/ElasticCache.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!