Unable to set service type, service name and log stream name

carlbergman commented 3 years ago

Problems

The variables AWS_EMF_SERVICE_TYPE and AWS_EMF_SERVICE_NAME are not included in the logged metric
The log stream in AWS_EMF_LOG_STREAM_NAME is ignored

Config

Running on ECS with Fargate (version 1.3.0)
The agent is in a sidecar
Using aws-embedded-metrics@2.0.1, Node 12

These are the container definitions:

ContainerDefinitions:
      # [...]
    - Name: app
      Image: redacted
      LogConfiguration:
          LogDriver: awslogs
          Options:
              awslogs-group: !Ref ContainerLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: app
      Environment:
          - Name: AWS_EMF_SERVICE_NAME
            Value: !Sub ${AWS::StackName}-app
          - Name: AWS_EMF_SERVICE_TYPE
            Value: "NodeJS-API"
          - Name: AWS_EMF_LOG_GROUP_NAME
            Value: !Ref ContainerLogGroup
          - Name: AWS_EMF_LOG_STREAM_NAME
            Value: metrics
          - Name: AWS_EMF_NAMESPACE
            Value: !Sub ${AWS::StackName}
          - Name: AWS_EMF_ENABLE_DEBUG_LOGGING
            Value: true
    - Name: agent
      Image: amazon/cloudwatch-agent:latest
      LogConfiguration:
          LogDriver: awslogs
          Options:
              awslogs-group: !Ref ContainerLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: agent
      Secrets:
          - Name: CW_CONFIG_CONTENT
            ValueFrom: !Ref CloudWatchAgentConfigArn

This is the agent config:

{
  "logs": {
    "metrics_collected": {
      "emf": {}
    }
  }
}

This is what the agent sidecar logs:

// Log stream: agent/agent/227b1b1f66744e318edb2d5e9bb57e2d

2020/11/09 16:31:03 I! 2020/11/09 16:31:03 E! ec2metadata is not available
--
2020/11/09 16:31:03 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
I! Detected the instance is ECS
2020/11/09 16:31:03 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directory2020/11/09 16:31:03 unable to scan config dir /etc/cwagentconfig with error: lstat /etc/cwagentconfig: no such file or directory
2020/11/09 16:31:03 Reading json config from from environment variable CW_CONFIG_CONTENT.
Valid Json input schema.
I! detect region from ecs
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
 
2020/11/09 16:31:03 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020-11-09T16:31:03Z I! Starting AmazonCloudWatchAgent 1.247346.0
2020-11-09T16:31:03Z I! Loaded inputs: socket_listener socket_listener
2020-11-09T16:31:03Z I! Loaded aggregators:
2020-11-09T16:31:03Z I! Loaded processors:
2020-11-09T16:31:03Z I! Loaded outputs: cloudwatchlogs
2020-11-09T16:31:03Z I! Tags enabled:
2020-11-09T16:31:03Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"", Flush Interval:1s
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on udp://[::]:25888
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on tcp://[::]:25888
2020-11-09T16:31:03Z I! [logagent] starting
2020-11-09T16:31:03Z I! [logagent] found plugin cloudwatchlogs is a log backend

This is what the app container logs:

// Log stream: app/app/227b1b1f66744e318edb2d5e9bb57e2d

Received default dimensions {
  LogGroup: 'redactedLogGroupName',
  ServiceName: 'my-service-name',
  ServiceType: 'NodeJS-API'
}
Sending {} events to socket. 1
opening connection with socket in state:  closed
TcpClient connected. { host: '0.0.0.0', port: 25888, protocol: 'tcp:' }
Write succeeded

However, the metric gets logged to another log stream (not the one in AWS_EMF_LOG_STREAM_NAME) and it does not include the AWS_EMF_SERVICE_TYPE or AWS_EMF_SERVICE_NAME.

// Log stream: arn_aws_ecs_eu-north-1_redactedAccountId_task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d

{
    "Endpoint": "POST /authentication/v5/login/email",
    "Method": "POST",
    "Path": "/authentication/v5/login/email",
    "StatusCode": 200,
    "IP": "redactedIp",
    "UserAgent": "Amazon CloudFront",
    "containerId": "ip-redactedIp.eu-north-1.compute.internal",
    "createdAt": "2020-11-09T16:31:04.591709491Z",
    "startedAt": "2020-11-09T16:31:05.226439Z",
    "image": "redactedAccountId.dkr.ecr.eu-north-1.amazonaws.com/redactedImage",
    "cluster": "arn:aws:ecs:eu-north-1:redactedAccountId:cluster/redactedClusterName",
    "taskArn": "arn:aws:ecs:eu-north-1:redactedAccountId:task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d",
    "_aws": {
        "Timestamp": 1604996779353,
        "LogGroupName": "redactedLogGroupName",
        "CloudWatchMetrics": [
            {
                "Dimensions": [
                    [
                        "Endpoint"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "Latency",
                        "Unit": "Milliseconds"
                    },
                    {
                        "Name": "Success",
                        "Unit": "Count"
                    }
                ],
                "Namespace": "redactedNamespace"
            }
        ]
    },
    "Latency": 1415.994263,
    "Success": 1
}

jaredcnance commented 3 years ago

I'll need to look into this a bit further. Any specific reason you want to hardcode the LogStream name? This can have scalability issues since a single LogStream is limited to 5 MB/s. Can you confirm the environment variables are available in your application process (e.g. console.log(process.env))? Are you calling setDimensions on your logger or putDimensions ?

Relevant tests:

https://github.com/awslabs/aws-embedded-metrics-node/blob/4b29d2eca26523d4c3ce430cf18ebdca2723cb88/src/config/__tests__/EnvironmentConfigurationProvider.test.ts#L49-L61

carlbergman commented 3 years ago

Thanks for the reply!

Any specific reason you want to hardcode the LogStream name? This can have scalability issues since a single LogStream is limited to 5 MB/s.

I wanted to make the log stream name similar to the log streams for the app (app/app/ecsTaskId) and agent (agent/agent/ecsTaskId) instead of the default name (arn_aws_ecs_eu-north-1_redactedAccountId_task/redactedClusterName/ecsTaskid).

That's when I found out that neither Configuration.logStreamName or AWS_EMF_LOG_STREAM_NAME had any effect. I was unaware of the scalability issues and will have to make a new decision regarding the name, but it would still be interesting to learn the proper way to set the log stream name.

Can you confirm the environment variables are available in your application process (e.g. console.log(process.env))?

I can confirm that the correct environment variables are logged when the application starts.

Are you calling setDimensions on your logger or putDimensions?

Thanks for pointing this out. I used setDimensions instead of putDimensions. By changing it to putDimensions the service name and service type are correctly added to the logged metric. Sorry for not realizing this before raising an issue. The problem with the log stream name still remains, though.

jaredcnance commented 3 years ago

but it would still be interesting to learn the proper way to set the log stream name

100% agree. I need to spend some time on this.

Sorry for not realizing this before raising an issue

No worries! Appreciate you opening the issue. I understand the confusion and open to hearing any suggestions as to how we can improve.

grandstairs commented 6 months ago

Yes, just looking at using this library in a lambda and finding that three dimensions are added by default.

"Dimensions": [
                    [
                        "LogGroup",
                        "ServiceName",
                        "ServiceType"
                    ]
],

The LogGroup and ServiceName were long random gibberish in each account I was using so I tried to change them but found that AWS_EMF_LOG_GROUP_NAME didn't seem to have any effect. I looked in the repo and it's not actually used anywhere I could find (except the test). AWS_EMF_SERVICE_NAME works though.

grandstairs commented 6 months ago

Update: The work around for me was to simply replace the default dimension set with a custom one and avoid trying to set the Log Group at all.

metricsLogger.setNamespace('FooBar');
metricsLogger.setDimensions({
        ServiceName: 'FooBar',
});

I would still like to know how it can be set.

awslabs / aws-embedded-metrics-node

Unable to set service type, service name and log stream name #66