[metrics]: support new namespace ApplicationSignals

[jianpingzhangBill]

AWS Applicaiton Signals Metrics fail to scrape.

Example: Here is my terraform code to create an exporter with query AppSignals

resource "helm_release" "cloudwatch_exporter" {

  name       = "cloudwatch-exporter"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "prometheus-cloudwatch-exporter"
  version    = "0.25.3"
  namespace  = data.aws_ssm_parameter.name_space.value
  wait       = "false"
  values = [<<-EOF
serviceMonitor:
  enabled: true
  timeout: 10s
# resources: 
#   limits:
#     cpu: 100m
#     memory: 128Mi
#   requests:
#     cpu: 100m
#     memory: 128Mi
serviceAccount:
  create: true
  name:
  # annotations:
  # Will add the provided map to the annotations for the created serviceAccount
  # e.g.
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${local.account_id}:role/prom-cloudwatch-exporter-oidc
    eks.amazonaws.com/sts-regional-endpoints: "true"

  automountServiceAccountToken: true

config: |-
  delay_seconds: 30
  period_seconds: 30
  debug: true
  metrics:
  # SELECT AVG(Latency) FROM AppSignals GROUP BY Operation, RemoteOperation, "HostedIn.K8s.Namespace",  Service, RemoteService
  - aws_namespace: AppSignals
    aws_metric_name: Latency
    aws_dimensions: [Operation, RemoteOperation, "HostedIn.K8s.Namespace",  Service, RemoteService]
    aws_statistics: [Average]

  - aws_dimensions:
    - CacheNodeId
    - CacheClusterId
    aws_metric_name: CPUUtilization
    aws_namespace: AWS/ElastiCache
    aws_statistics:
    - Average
  - aws_dimensions:
    - CacheNodeId
    - CacheClusterId
    aws_metric_name: CPUCreditBalance
    aws_namespace: AWS/ElastiCache
    aws_statistics:
    - Average
  - aws_dimensions:
    - CacheNodeId
    - CacheClusterId
    aws_metric_name: FreeableMemory
    aws_namespace: AWS/ElastiCache
    aws_statistics:
    - Average

  EOF

  ]
  depends_on = [
    shell_script.init
  ]
}

resource "aws_iam_policy" "cloudwatch_exporter" {
  name   = "prom-cloudwatch-exporter-oidc-policy"
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey*",
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:ReEncrypt*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "tag:GetResources",
                "cloudwatch:GetMetricData",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics",
                "apigateway:GET",
                "aps:ListWorkspaces",
                "autoscaling:DescribeAutoScalingGroups",
                "dms:DescribeReplicationInstances",
                "dms:DescribeReplicationTasks",
                "ec2:DescribeTransitGatewayAttachments",
                "ec2:DescribeSpotFleetRequests",
                "shield:ListProtections",
                "storagegateway:ListGateways",
                "storagegateway:ListTagsForResource"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}
EOF
}

resource "aws_iam_role" "cloudwatch_exporter_service_role" {
  name = "prom-cloudwatch-exporter-oidc"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Federated = "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn}"
        }
        Condition = {
          StringEquals = {
            "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:sub": "system:serviceaccount:${data.aws_ssm_parameter.name_space.value}:cloudwatch-exporter-prometheus-cloudwatch-exporter"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "extra_policy" {
  policy_arn = aws_iam_policy.cloudwatch_exporter.arn
  role       = aws_iam_role.cloudwatch_exporter_service_role.name
}

[matthiasr]

The documentation says that the namespace is ApplicationSignals. It also lists the valid combinations of dimensions – it seems that you always need to include Environment for each Service? There is also no mention of HostedIn.K8s.Namespace anywhere.

What do you get with aws cloudwatch list-metrics --namespace AWS/ApplicationSignals? Can you share an invocation of aws cloudwatch get-metric-statistics that gets you the data you want?

[matthiasr]

In general, the exporter does not need explicit support for new namespaces, but we do need to work out how exactly to teach it about them. We then add that to the examples.