DBM monitoring, and other label config not support on ECS FARGATE. Allow setting via ENV.

[pieterza]

The documentation for ECS/Docker and RDS DBM contains only the use of labels to set the DBM config: https://docs.datadoghq.com/database_monitoring/setup_mysql/aurora/?tab=docker

  -l com.datadoghq.ad.check_names='["mysql"]' \
  -l com.datadoghq.ad.init_configs='[{}]' \
  -l com.datadoghq.ad.instances='[{
    "dbm": true,
    "host": "<AWS_INSTANCE_ENDPOINT>",
    "port": 3306,
    "username": "datadog",
    "password": "<UNIQUEPASSWORD>"
  }]' \

This relies on /var/run/docker.sock being mounted as read only, for the agent to be able to read it's own labels. This is not supported on 'serverless' container platforms such as ECS FARGATE, and others which have limited access to the underlying host.

Please allow us to set the DBM config via ENV perhaps, as I have tested by passing in a custom datadog config that DBM is working even on FARGATE as long as we do not need to read labels or otherwise touch the docker socket.

In the mean time, I'm forced to building a config (example below) and passing it into the container as a hacky workaround for anyone else with this issue.

Init_config:

instances:
  - dbm: true
    host: '<AWS_INSTANCE_ENDPOINT>'
    port: 3306
    username: datadog
    password: '<YOUR_CHOSEN_PASSWORD>' # from the CREATE USER step earlier

[AndriiChuzhynov]

Hi, @pieterza try to add ECS_FARGATE=true environment variable. In this case, the agent will be able to get labels.

[lifeofguenter]

Setting com.datadoghq.ad.* as label and ECS_FARGATE=true on the main container (not the datadog-agent sidecar) worked for us for our use cases.

Maybe this is something specific to DBM?

[ggorge-etiqa]

Utilizing labels to transmit connection details such as the password within an ECS Fargate setup entails exposing them in plaintext within the task-definition.json file. To enhance security, a preferable approach would involve configuring these sensitive parameters as environment variables, allowing for encryption measures to be implemented.

[jcave-relay]

Hi. I have a use case where I have a lambda function that uses the rds instance that I want to monitor. I created a Fargate cluster with the agent as the sole service/task running in the cluster and setting ECS_FARGATE=true on that container and the labels are not working for me. Here's my container definition (in terraform):

{
      name      = "dd-agent"
      image     = "public.ecr.aws/datadog/agent:latest"
      essential = true
      portMappings = [
        {
          containerPort = 8125
          hostPort      = 8125
          protocol      = "udp"
        },
        {
          containerPort = 8126
          hostPort      = 8126
          protocol      = "tcp"
        }
      ]
      environment = [
        {
          name  = "ENV"
          value = terraform.workspace
        },
        {
          name  = "DD_TAGS",
          value = "env:${terraform.workspace} environment:${terraform.workspace} region:${var.region}"
        },
        {
          name  = "ECS_FARGATE",
          value = "true"
        }
      ]
      secrets = [
        {
          name      = "DD_API_KEY"
          valueFrom = data.aws_secretsmanager_secret.dd_api_key.arn
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.dd_agent.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "dd-agent"
        }
      }
      dockerLabels = {
        "com.datadoghq.ad.checks" = jsonencode({
          postgres = {
            init_config = [{}]
            instances = [
              {
                dbm      = true
                host     = aws_db_instance.rds.address
                port     = 5432
                username = "datadog"
                aws = {
                  instance_endpoint = aws_db_instance.rds.address,
                  region            = var.region
                  managed_authentication = {
                    enabled = true
                  }
                }
              }
            ]
          }
        })
      }
    }

[jcave-relay]

Hi. I have a use case where I have a lambda function that uses the rds instance that I want to monitor. I created a Fargate cluster with the agent as the sole service/task running in the cluster and setting ECS_FARGATE=true on that container and the labels are not working for me. Here's my container definition (in terraform):

{
      name      = "dd-agent"
      image     = "public.ecr.aws/datadog/agent:latest"
      essential = true
      portMappings = [
        {
          containerPort = 8125
          hostPort      = 8125
          protocol      = "udp"
        },
        {
          containerPort = 8126
          hostPort      = 8126
          protocol      = "tcp"
        }
      ]
      environment = [
        {
          name  = "ENV"
          value = terraform.workspace
        },
        {
          name  = "DD_TAGS",
          value = "env:${terraform.workspace} environment:${terraform.workspace} region:${var.region}"
        },
        {
          name  = "ECS_FARGATE",
          value = "true"
        }
      ]
      secrets = [
        {
          name      = "DD_API_KEY"
          valueFrom = data.aws_secretsmanager_secret.dd_api_key.arn
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.dd_agent.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "dd-agent"
        }
      }
      dockerLabels = {
        "com.datadoghq.ad.checks" = jsonencode({
          postgres = {
            init_config = [{}]
            instances = [
              {
                dbm      = true
                host     = aws_db_instance.rds.address
                port     = 5432
                username = "datadog"
                aws = {
                  instance_endpoint = aws_db_instance.rds.address,
                  region            = var.region
                  managed_authentication = {
                    enabled = true
                  }
                }
              }
            ]
          }
        })
      }
    }

In case anyone stumbles upon this in the future.. I reached out to Datadog support about this and was sent to https://docs.datadoghq.com/integrations/faq/integration-setup-ecs-fargate/?tab=rediswebui. I followed this doc and am now all set with labels.

The difference between this doc and what I had originally done after following the datadog DBM setup docs, is the labels com.datadoghq.ad.instances, com.datadoghq.ad.check_names, com.datadoghq.ad.init_configs vs just com.datadoghq.ad.checks in the datadog DBM setup docs.