aws / aws-app-mesh-examples

AWS App Mesh is a service mesh that you can use with your microservices to manage service to service communication.
MIT No Attribution
856 stars 397 forks source link

[BUG] Metrics related to Outbound Traffic are not exported after enabling the appmesh metric extension #587

Closed ssun3 closed 4 months ago

ssun3 commented 4 months ago

Describe the bug Envoy sidecar container configured with APPMESH_METRIC_EXTENSION_VERSION=1 does not export metrics related to outbound traffic (e.g. envoy.appmesh.TargetProcessedBytes, envoy.appmesh.HTTPCode_Target_2XX_Count ...)

Platform ECS

To Reproduce Steps to reproduce the behavior:

  1. Launch an app-mesh enabled ecs task

Sample anonymised task config:

anonymised task config:
```json
{
    "taskDefinitionArn": "arn:aws:ecs:xxx:xxx:task-definition/xxx:35",
    "containerDefinitions": [
        {
             ...app_container_def
        },
        {
            "name": "envoy",
            "image": "xxx.dkr.ecr.xxx.amazonaws.com/ecr-public/appmesh/aws-appmesh-envoy:v1.27.2.0-prod",
            "cpu": 0,
            "memory": 500,
            "portMappings": [],
            "essential": true,
            "environment": [
                {
                    "name": "APPMESH_METRIC_EXTENSION_VERSION",
                    "value": "1"
                },
                {
                    "name": "ENABLE_ENVOY_STATS_TAGS",
                    "value": "1"
                },
                {
                    "name": "APPMESH_RESOURCE_ARN",
                    "value": "arn:aws:appmesh:eu-west-2:xxx:mesh/default/virtualNode/app"
                },
                {
                    "name": "ENABLE_ENVOY_XRAY_TRACING",
                    "value": "1"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "1337",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/aws/ecs/xxx",
                    "awslogs-region": "xxx",
                    "awslogs-stream-prefix": "envoy"
                }
            },
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -s http://localhost:9901/server_info | grep state | grep -q LIVE"
                ],
                "interval": 5,
                "timeout": 2,
                "retries": 3,
                "startPeriod": 10
            }
        }
    ],
    "family": "xxx",
    "taskRoleArn": "arn:aws:iam::xxx:role/xxx",
    "executionRoleArn": "arn:aws:iam::xxx:role/xxx",
    "networkMode": "awsvpc",
    "revision": 35,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
         ...
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "proxyConfiguration": {
        "type": "APPMESH",
        "containerName": "envoy",
        "properties": [
                ... 
        ]
    },
    "registeredAt": "xxx",
    "registeredBy": "xxx",
    "tags": []
}
  1. Exec into the ecs service and curl the envoy stats endpoint:

Metrics related to Inbound Traffic are available:

Screenshot 2024-02-21 at 14 31 01 Screenshot 2024-02-21 at 15 05 23

But Metrics related to Outbound Traffic are not:

Screenshot 2024-02-21 at 14 30 54 Screenshot 2024-02-21 at 14 32 35

Expected behavior Expected to see the metrics related to outbound traffic

ssun3 commented 4 months ago

The issues are fixed with the latest release of v1.27.3.0. related issue 448