Custom metrics on fargate not working

[cvele]

Service is launched on AWS and is composed out of 3 containers (task definition below), amazon/aws-for-fluent-bit, datadog/agent:7.22.1 and the PHP 7.3 application which includes datadog client. Service is configured using awsvpc networking mode.

Datadog agent host and port are not set in PHP client, but are passed via environment variables. Service is up and operational, with no apparent issues. Logs are working, containers are visible in infrastructure section of datadog application.

However, custom metrics are not visible in datadog application. The same PHP application is being deployed to EC2 instance with datadog-agent as a service and custom metrics sent from that context are being displayed in datadog application.

At this point I am very confused and lost. I'm not able to find any solutions to my issue.

Below you can find my task definition.

        "essential": true,
        "image": "amazon/aws-for-fluent-bit:latest",
        "name": "log_router",
        "firelensConfiguration": {
            "type": "fluentbit",
            "options": {
                "enable-ecs-log-metadata": "true",
                "config-file-type": "file",
                "config-file-value": "/fluent-bit/configs/parse-json.conf"
            }
        }
    },
    {
        "name": "${app_name}",
        "image": "${app_image}",
        "portMappings": [],
        "essential": true,
        "entryPoint": ["sh","-c"],
        "command": ["${command}"],
        "logConfiguration": {
            "logDriver": "awsfirelens",
            "options": {
                "Name": "datadog",
                "Host": "http-intake.logs.datadoghq.com",
                "TLS": "on",
                "dd_service": "${dd_service}",
                "dd_source": "${dd_source}",
                "dd_tags": "${dd_tags}",
                "provider": "ecs"
            },
            "secretOptions": [{
                "name": "apikey",
                "valueFrom": "${datadog_apikey}"
            }]
        },
        "environment": [
            {
                "name": "ECS_AVAILABLE_LOGGING_DRIVERS",
                "value": "[\"json-file\", \"awslogs\"]"
            },
            {
                "name": "SIS_ENV",
                "value": "${environment}"
            },
            {
                "name": "ENVIRONMENT",
                "value": "${environment}"
            },
            {
                "name": "DD_AGENT_HOST",
                "value": "localhost"
            },
            {
                "name": "DD_DOGSTATSD_PORT",
                "value": "8126"
            }
        ],
        "secrets": [
            {
                "valueFrom": "${AWS_ACCESS_KEY_ID}",
                "name": "AWS_ACCESS_KEY_ID"
            },
            {
                "valueFrom": "${AWS_SECRET_ACCESS_KEY}",
                "name": "AWS_SECRET_ACCESS_KEY"
            },
            {
                "name": "DD_API_KEY",
                "valueFrom": "${datadog_apikey}"
            }
        ]
    },
    {
        "name": "datadog-agent",
        "image": "datadog/agent:7.22.1",
        "essential": true,
        "portMappings": [ 
            { 
               "containerPort": 8125,
               "protocol": "udp"
            },
            { 
               "containerPort": 8126,
               "protocol": "udp"
            },
            { 
                "containerPort": 8125,
                "protocol": "tcp"
            }
        ],
        "logConfiguration": {
            "logDriver": "awsfirelens",
            "options": {
                "Name": "datadog",
                "Host": "http-intake.logs.datadoghq.com",
                "TLS": "on",
                "dd_service": "${dd_service}",
                "dd_source": "datadog.agent.log",
                "dd_tags": "${dd_tags}",
                "provider": "ecs"
            },
            "secretOptions": [{
                "name": "apikey",
                "valueFrom": "${datadog_apikey}"
            }]
        },
        "secrets": [{
            "name": "DD_API_KEY",
            "valueFrom": "${datadog_apikey}"
        }],
        "environment": [{
                "name": "DD_LOGS_ENABLED",
                "value": "true"
            },
            {
                "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
                "value": "true"
            },
            {
                "name": "DD_ENABLE_METADATA_COLLECTION",
                "value": "true"
            },
            {
                "name": "DD_PROCESS_AGENT_ENABLED",
                "value": "true"
            },
            {
                "name": "DD_AC_EXCLUDE",
                "value": "name:datadog-agent"
            },
            {
                "name": "ECS_FARGATE",
                "value": "true"
            },
            {
                "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
                "value": "true"
            },
            {
                "name": "DD_DOGSTATSD_ORIGIN_DETECTION",
                "value": "true"
            },
            {
                "name": "DD_DOGSTATSD_STATS_ENABLE",
                "value": "true"
            },
            {
                "name": "DD_DOGSTATSD_PORT",
                "value": "8126"
            }
        ]
    }
]```

[cvele]

Out of desparation, I've also opened the DD ports by assigning security group to the service:

    description = "DataDogStatsD UDP Port"
    from_port   = 8125
    to_port     = 8125
    protocol    = "udp"
    cidr_blocks = [data.aws_vpc.this.cidr_block]
  }

  ingress {
    description = "DataDogStatsD UDP Port"
    from_port   = 8126
    to_port     = 8126
    protocol    = "udp"
    cidr_blocks = [data.aws_vpc.this.cidr_block]
  }

  ingress {
    description = "DataDogStatsD TCP Port"
    from_port   = 8125
    to_port     = 8125
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.this.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

I've also attempted using 8125 as DD_DOGSTATSD_PORT and 127.0.0.1 as DD_AGENT_HOST.

[jens]

Dogstatsd listens by default on UDP port 8125 not 8126. TCP port 8126 is for APM

[gunjan-chauhan-dev]

Did you get this to work, facing same issue with ECS FARGATE awsvpc

[jakub-qg]

To all the souls lost here:

            {
              name  = "DD_USE_DOGSTATSD"
              value = "true"
            },