Bring uniformity in `aws.dimensions.*` fields mapping

[zmoog]

Situation

Elastic offers multiple integrations to collect metrics from CloudWatch in the AWS integration package.

Some are "specialized" and only collect metrics from one AWS service, like the EC2, ECS, Redshift, and others. However, one "generic" integration allows users to write the integration configs to collect metrics from any AWS service. The generic integration is excellent for collecting metrics from services we have not supported.

All integrations (specialized and generic), collect metric dimensions as aws.dimentions.* fields.

Problem

Unfortunately, not all metrics integrations use the exact mapping for the aws.dimentions field.

Integration	Type	Mapping for aws.dimentions
Collect EC2 logs from CloudWatch	Specialized	object
Collect metrics from CloudWatch	Generic	flattened

With different mappings, it's hard for users to correlate dimensions and build dashboards.

Solution

Work in progress.

We need to identify:

• What is the best mapping to represent AWS metric dimensions
• Describe a custom mapping to work around the issue in the short term (if it exists).
• Identify a mid-long term solution to bring uniformity to dimensions field mapping.

References

Related:

• A large migration to flattened: https://github.com/elastic/integrations/pull/8115/files

[zmoog]

For a short-term solution for users who want to map aws.dimensions as an object of keywords, I tried defining a custom dynamic template in the generic integration.

  - name: dimensions.*
    type: object
    object_type: keyword
    object_type_mapping_type: "*"

However, the aws.dimensions takes precedence.