datadog_integration_aws account_specific_namespace_rule is not working

[n0ts]

Hi, datadog_integration_aws account_specific_namespace_rule is not working.

Terraform Version

0.11.13

Provider Version

1.8.0

Affected Resource(s)

• datadog_integration_aws

Terraform Configuration Files

data "aws_caller_identity" "identity" {}

resource "datadog_integration_aws" "datadog" {
  account_id = "${data.aws_caller_identity.identity.account_id}"
  role_name  = "datadog"

  filter_tags = [
    "Name:app-instance",
  ]

  account_specific_namespace_rules = {
    "application_elb" = true
    "auto_scaling" = true
    "billing" = true
    "budgeting" = true
    "cloudwatch_events" = true
    "cloudwatch_logs" = true
    "codebuild" = true
    "collect_custom_metrics" = true
    "crawl_alarms" = true
    "ddos_protection" = true
    "ebs" = true
    "ec2" = true
    "ec2api" = true
    "ec2spot" = true
    "ecs" = true
    "efs" = true
    "elasticache" = true
    "elb" = true
    "kms" = true
    "lambda" = true
    "nat_gateway" = true
    "network_elb" = true
    "rds" = true
    "redshift" = true
    "route53" = true
    "s3" = true
    "ses" = true
    "sns" = true
    "sqs" = true
    "waf" = true
    "xray" = true
  }
}

Expected Behavior

Datadog AWS integration Limit metric collection by AWS Service is set be default. Screenshot is here. https://gyazo.com/d5e75a3694815645cda9acddab39eaaa

Actual Behavior

Checked Billing, EBS, EC2` and more.

https://api.datadoghq.com/api/v1/integration/aws/available_namespace_rules?api_key= endpoint available billing etc...

[
  "api_gateway",
  "application_elb",
  "appsync",
  "auto_scaling",
  "billing",
  "budgeting",
...

Steps to Reproduce

1. terraform apply
2. Open datadog aws integration details page https://app.datadoghq.com/account/settings#integrations/amazon_web_services

Regrards,

[nmuesch]

Hey @n0ts Thanks for reaching out. I've been looking into this issue and have some updates.

The Limit Metric Collection section from your screenshot is a list of namespaces that would apply globally across the AWS accounts you've setup IF the specific accounts don't have Special Rules in place. These special rules are the ones you've setup in your Terraform config. On the Integration tile, you should see X Special Rules near the account id.

I've made a note with the appropriate team to see if we can get this clarified in the UI. That being said, this shouldn't have any impact on the metrics coming into your account.

I'll close this out for now but let me know if you have further impact here!

[aalimovs]

@nmuesch I'm having issues with account_specific_namespace_rules too, quite confusing what's going on.

I have a full list of all account_specific_namespace_rules's and all except 2 are set to false. Applying the change creates all possible AWS integrations that I did not ask to. See below:

[finferflu]

@nmuesch I'm resurrecting an old thread, but the above question was never addressed. I, too, have disabled most integrations, but I'm finding that they're all enabled when I visit the Datadog integrations page, as per @aalimovs's screenshot. Am I missing something?

Thanks!

---

Replying to my own question here (and the one above mine), it looks like the expectation of what this does is wrong, as account_specific_namespace_rules creates a filter on the metric collection, rather than on which integrations are enabled, as per this screenshot. It would be nice to have a way to disable all those integrations, though!

[nmuesch]

Hey, I apologize for missing the above question. @aalimovs and @finferflu. I'll go ahead and re-open this one for now

[stueccles]

+1 on this issue

I'll also add that even if you disable the integrations manually, updating the terraform resource causes them to be re-enabled.

[gavinclarkeuk]

Just done a bunch of testing around this as certain integrations were getting enabled causing our datadog bills to skyrocket.

We found that not specifying the namespace rule for the EMR integration would cause the EMR integration to get installed whenever there was another change to the datadog_integration_aws resource (e.g. tags). If we explicitly set emr to false then the EMR integration won't get installed, but it won't get uninstalled either, we have to so that manually.

Seems like a provider issue to me, as ultimately it isn't declarative - the final state depends on the starting state which just shouldn't happen with terraform.

[iniinikoski]

I've noticed the same thing @gavinclarkeuk here, which also has exploded our CW Metrics invoicing. I guess we can send those invoices to Datadog 😃

I think we'll reach out to our customer success team here but will also work on a fix by just disabling every current darn namespace we are not using.

Datadog definitely needs to change this for better somehow, as every AWS customer has the same issue.