Custom global dimensions

[okgolove]

Hi! Thank you for the software. I'd like to ask what was the reason to forbid specifying custom global dimensions? I would like to add a global custom dimension environment name: <env name> for ALL metrics from cloudwatch agent for example. Yes, I can add it manually for most of agent inputs, but I can't do this for statsd and collectd, for example.

I'm talking about this part:

append_dimensions – Optional. Adds Amazon EC2 metric dimensions to all metrics collected by the agent. The only supported key-value pairs are shown in the following list. Any other key-value pairs are ignored.

"ImageID":"${aws:ImageId}" sets the instance's AMI ID as the value of the ImageID dimension.
"InstanceId":"${aws:InstanceId}" sets the instance's instance ID as the value of the InstanceID dimension.
"InstanceType":"${aws:InstanceType}" sets the instance's instance type as the value of the InstanceType dimension.
"AutoScalingGroupName":"${aws:AutoScalingGroupName}" sets the instance's Auto Scaling group name as the value of the AutoScalingGroupName dimension.

May be you are planning to add this feature or you can suggest any workaround. Thanks!

[pxaws]

There is no reason to forbid specifying custom global dimensions. Existing variables in the config like "${aws:ImageId}" are not environment variables. Cloudwatch agent gets their values by querying the ec2 metadata service.

To support the feature you mentioned, we would like to know more about your use case. For example, are you running cloudwatch agent in container or not?

For container environments, it might be easy to specify the environmental variables through docker file. When cloudwatch agent runs, it can get the values for those environmental variables.

For non-container environments, we would like to know how do you specify the environmental variables so that cloudwatch agent can pick them up? (You know cloudwatch agent might run in a different shell from the one in which those environmental variables are set up)

[okgolove]

Sorry, I must described my thoughts wrong. I'm no talking about environment variables, I'm talking about environment name (I meant staging/dev/UAT/etc). My goal is specifying global dimension instead of append dimensions per metric (MySuperCustomDimension is an example):

{
    "metrics": {
        "append_dimensions": {
            "MySuperCustomDimension": "everything_i_want",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}",
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
        },
        "metrics_collected": {
            "collectd": {},
            "cpu": {
                "resources": [
                    "*"
                ],
                "measurement": [
                    {
                        "name": "cpu_usage_idle",
                        "rename": "CPU_USAGE_IDLE",
                        "unit": "Percent"
                    },
                    {
                        "name": "cpu_usage_nice",
                        "unit": "Percent"
                    },
                    "cpu_usage_guest"
                ],
                "totalcpu": false,
                "metrics_collection_interval": 10,
                "append_dimensions": {
                    "test": "test1",
                    "date": "2017-10-01"
                }
            },
            "netstat": {
                "measurement": [
                    "tcp_established",
                    "tcp_syn_sent",
                    "tcp_close"
                ],
                "metrics_collection_interval": 60
            },
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "resources": [
                    "*"
                ],
                "drop_device": true
            },
            "processes": {
                "measurement": [
                    "running",
                    "sleeping",
                    "dead"
                ]
            }
        },
        "aggregation_dimensions": [
            [
                "AutoScalingGroupName"
            ],
            [
                "InstanceId",
                "InstanceType"
            ],
            []
        ]
    }
}

[matan129]

Hi, I'm interested in this specific use case. Are there any plans to implement it?

[matan129]

For the statsd specifically, I've found a workaround using Telegraf-style templates. These are actually already implemented in CW Agent, but for some reason were not enabled in the config.

I've submitted a PR to fix that: #260

Usage:

{
  "statsd": {
    "templates": [
      "measurement.* MY_CUSTOM_TAG=value
    ]
  }
}

[dxciberaws]

Need this for meaningful way of declaring alarms in CW on agent metrics

[mzraly]

Use case 1: We want to track metrics separately for different service types, but AutoScalingGroupName does not work for us because our blue-green deployments create new autoscaling groups for each deployment. We don't want to rely on finding resources by tags like Metrics Explorer because that only finds currently active resources. If we want to see how metrics evolved across releases we need to know all the relevant autoscaling group names. We'd rather use a search expression, which has the advantage of finding metrics for terminated resources and working from any region.

Use case 2: We want to use custom dimensions in dynamic labels in our CloudWatch widgets.

Very interested in this feature being implemented.

[jlosito]

I have a use case where I'm tagging EC2 instances with a tag in order to determine whether I want to enable alerting on this instance. I want to be able to push that EC2 tag as a dimension in order to filter on the instances that have the alerting EC2 tag set to true.

With the current limitations of the agent, I believe the only workaround would be to query the metrics I am looking for with a custom script and push the extra dimension apart of the script. Is that correct? Since the other dimensions are being queried from the metadata endpoint, would it be possible to add tags to the list?

[ksmalik1]

I need this enhancement for collectd.

My use case is to tag the metrics with the name/role of the server publishing the metrics. ${aws:InstanceId} doesn't work because it changes every time the EC2 instance is recreated. Our deployment system can create EC2 instances with name/role as EC2 Tags and also set it as an environment variable for system.d CWAgent service. But there is no way to specify the tags or environment variables as `appended dimensions" to the collectd section or the global section of the CWAgent config.

[jordanade]

Why is it so hard to gather RAM usage organized by environment name? I tried this and it doesn't work:

          "append_dimensions": {
            "APP_ENV": "`{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "APP_ENV"}}`",
            "InstanceId": "${aws:InstanceId}"
          },
          "aggregation_dimensions": [
            ["APP_ENV"],
            ["APP_ENV", "InstanceId"]
          ],