AWS::Logs::MetricFilter does not accept dimensions

[srinivas-vangari]

The following announcement makes dimensions available for metric filters.

https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-cloudwatch-logs-announces-dimension-support-for-metric-filters/

However, it appears to be missing when specifying the resource through CloudFormation. Getting the following error on stack events console.

Encountered unsupported property Dimensions

The Python SDK documentation correctly shows this as available, but missing in CloudFormation docs for CloudWatchLogs. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.put_metric_filter

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html

[srinivas-vangari]

This is a duplicate of https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/419

[arnulfojr]

Any updates?

[waifa]

Working on this feature request. Support should be available shortly.

[waifa]

Deployment beginning shortly. I will update here once completed.

[henhal]

@waifa Any updates on this? 🙏

[gshpychka]

@waifa can you advise on the status of this?

[waifa]

The deployment from February failed unfortunately. We are currently working on a fix, thanks for your patience!

[samjarman]

@waifa Hey there! Curious if there are any updates on this? Thanks!

[KirtiKunalShah]

We have been working on issues related to deployment and traffic migration. Stay tuned to update which I should be able to provide by End of July

[mbarneyjr]

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

[jakob-keller]

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

I just checked in eu-central-1: It's not live as of now. However, it's possible to use the Unit property which is not yet covered in the CloudFormation documentation.

[kgston]

The documentation description for the Dimensions property is a little weird, but I've managed to successfully use it in ap-northeast-1

[stevielb]

@KirtiKunalShah You mentioned there might be an update around this time. Anything you can share with us on current progress?

[jyotimahapatra]

@waifa Can you help with a sample of how dimensions work? Looks like this is available in us-west-2 and i can consume it. However, i get this error in Cfn Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter" (RequestToken: abc41535-8432-1c18-e4ad-45438e510ff2, HandlerErrorCode: InvalidRequest)

Can you help with an example ?

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "CloudFormation template configuring AWS EKS Dataplane AZ metrics log groups",
    "Resources": {
        "KubeApiServerAzHealthzAPAusiw1az1MetricFilter1": {
            "Type": "AWS::Logs::MetricFilter",
            "Properties": {
                "LogGroupName": "abc",
                "FilterPattern": "{ $.xyz < 0.9 && $.AZ = usiw1-az1 }",
                "MetricTransformations": [{
                    "MetricValue": "1",
                    "MetricNamespace": "something",
                    "MetricName": "something",
                    "Dimensions": [{
                      "Key": "ab",
                      "Value": "de"
                    }]
                }],
            }
        }
    }
}

[waifa]

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

[waifa]

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

I will close this issue once it is available in all regions.

[jakob-keller]

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

As of now, it is not available in eu-central-1:

CREATE_FAILED - Encountered unsupported property Dimensions

[rene-kochen]

It used to work in eu-central-1

Now our stacks no longer deploy:

12:40:53 | CREATE_FAILED | AWS::Logs::MetricFilter | MetricFilter Encountered unsupported property Dimensions

[waifa]

Sincere apologies from AWS for this inconsistency. A mistake on our end caused the feature to be rolled back in eu-central-1 for a week and a couple other regions for a short period of time.

[kathampy]

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

Can you support arbitrary values for dimension name? In my case the dimension name is not present in the log event and I need to specify it manually for the filter.

[waifa]

AWS::Logs::MetricFilter now accepts dimensions!

[msambol]

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in the log and thus is not extracted by the filterPattern.

[gordonpn]

@msambol

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in log and thus is not extracted by the filterPattern.

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

[r-heimann]

Are you guys 100% sure that this is working in eu-central-1?

Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter.
The specified filter pattern does not support dimensions
(Service: CloudWatchLogs, Status Code: 400, Request ID: a2271e5d-6464-4bfc-84db-XXXXXXXX)"
(RequestToken: 7b3cce6f-d6f0-8718-beb2-XXXXXXXX, HandlerErrorCode: InvalidRequest)

Example:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          Dimensions:
            - Key: "ClusterName"
              Value: !Ref ECSCluster
            - Key: "ServiceName"
              Value: !GetAtt ECSService.Name
          MetricValue: 1
          Unit: "Count"

This is working, by the way:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          MetricValue: 1
          Unit: "Count"

[gordonpn]

@r-heimann

Are you guys 100% sure that this is working in eu-central-1?

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#logs-metric-filters-dimensions:~:text=publish%20dimensions%20with%20metrics%20that%20JSON%20and%20space%2Ddelimited%20log%20events%20generate

[r-heimann]

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

I see, i have no idea why such a limitation exists in the first place but i'll try to get it working using your link.

[imangd]

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

@gordonpn it would be great to include this feature on roadmap for the use cases it can support :)

[alancnet]

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

[gordonpn]

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

@alancnet I agree with you and this is also something I raised internally, which is also pending prioritization.

[brentcetinich]

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

Raised this issue since this issue is not solved still.

[koenigstag]

Any updates on this?