NoAwsRegion: Cannot fetch region variable from config file

[rcdailey]

I followed the instructions to set up the AWS otel collector in a docker container. I do not use the auth environment variables, so I instead mounted my .aws directory. I'm using Docker Compose on Windows 10. See the compose YAML below.

otel:
    image: amazon/aws-otel-collector
    command: ["--config=/config/otel-collector-config.yml"]
    networks: [app_network]
    environment:
      - AWS_PROFILE
    volumes:
      - ./otel-config:/config:ro
      - $HOME/.aws:/root/.aws:ro

I have another service in this compose stack that houses my actual microservice (C# webapi project). It shares the same bridge network. That particular project uses the AWS .NET SDKs, and we also mount the .aws directory there and it works.

I did also try adding the AWS environment variables but that didn't seem to change the outcome. I'm not sure what the error is trying to tell me.

Here is the contents of my .aws directory:

config:

[profile MyProfile]
region = us-east-1
output = json

credentials:

[MyProfile]
aws_access_key_id=snip
aws_secret_access_key=snip
aws_session_token=snip

In my Windows environment, I have the AWS_PROFILE environment variable set to MyProfile.

Here is my collector config:

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:

exporters:
  logging:
    verbosity: detailed
  awsemf:
    log_group_name: snip
    log_stream_name: snip
    namespace: snip
    dimension_rollup_option: 1
    log_retention: 60

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

The logs I get when I do docker compose up are below. What do these errors mean and what is causing them?

docker-otel-1       | 2023/02/08 20:36:19 ADOT Collector version: v0.26.0
docker-otel-1       | 2023/02/08 20:36:19 found no extra config, skip it, err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory
docker-otel-1       | 2023-02-08T20:36:19.342Z  info    service/telemetry.go:90 Setting up own telemetry...
docker-otel-1       | 2023-02-08T20:36:19.343Z  info    service/telemetry.go:116        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
docker-otel-1       | 2023-02-08T20:36:19.371Z  error   awsutil@v0.70.0/conn.go:146     Unable to retrieve the region from the EC2 instance     {"kind": "exporter", "data_type": "metrics", "name": "awsemf", "error": "EC2MetadataRequestError: failed to get EC2 instance identity document\ncaused by: EC2MetadataError: failed to make EC2Metadata request\nconnecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.\n\tstatus code: 403, request id: "}
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil.GetAWSConfigSession
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil@v0.70.0/conn.go:146
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.newEmfPusher
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/emf_exporter.go:75
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.newEmfExporter
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/emf_exporter.go:102
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.createMetricsExporter
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/factory.go:62
docker-otel-1       | go.opentelemetry.io/collector/exporter.CreateMetricsFunc.CreateMetricsExporter
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/exporter/exporter.go:122
docker-otel-1       | go.opentelemetry.io/collector/exporter.(*Builder).CreateMetrics
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/exporter/exporter.go:260
docker-otel-1       | go.opentelemetry.io/collector/service.buildExporter
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/pipelines.go:361
docker-otel-1       | go.opentelemetry.io/collector/service.buildPipelines
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/pipelines.go:248
docker-otel-1       | go.opentelemetry.io/collector/service.(*Service).initExtensionsAndPipeline
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/service.go:199
docker-otel-1       | go.opentelemetry.io/collector/service.New
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/service.go:114
docker-otel-1       | go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/otelcol/collector.go:163
docker-otel-1       | go.opentelemetry.io/collector/otelcol.(*Collector).Run
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/otelcol/collector.go:205
docker-otel-1       | main.newCommand.func1
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:122
docker-otel-1       | github.com/spf13/cobra.(*Command).execute
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:916
docker-otel-1       | github.com/spf13/cobra.(*Command).ExecuteC
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:1044
docker-otel-1       | github.com/spf13/cobra.(*Command).Execute
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:968
docker-otel-1       | main.runInteractive
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:84
docker-otel-1       | main.run
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main_others.go:42
docker-otel-1       | main.main
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:77
docker-otel-1       | runtime.main
docker-otel-1       |   runtime/proc.go:250
docker-otel-1       | 2023-02-08T20:36:19.371Z  error   awsutil@v0.70.0/conn.go:156     Cannot fetch region variable from config file, environment variables and ec2 metadata.   {"kind": "exporter", "data_type": "metrics", "name": "awsemf"}
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil.GetAWSConfigSession
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/awsutil@v0.70.0/conn.go:156
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.newEmfPusher
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/emf_exporter.go:75
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.newEmfExporter
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/emf_exporter.go:102
docker-otel-1       | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter.createMetricsExporter
docker-otel-1       |   github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter@v0.70.0/factory.go:62
docker-otel-1       | go.opentelemetry.io/collector/exporter.CreateMetricsFunc.CreateMetricsExporter
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/exporter/exporter.go:122
docker-otel-1       | go.opentelemetry.io/collector/exporter.(*Builder).CreateMetrics
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/exporter/exporter.go:260
docker-otel-1       | go.opentelemetry.io/collector/service.buildExporter
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/pipelines.go:361
docker-otel-1       | go.opentelemetry.io/collector/service.buildPipelines
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/pipelines.go:248
docker-otel-1       | go.opentelemetry.io/collector/service.(*Service).initExtensionsAndPipeline
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/service.go:199
docker-otel-1       | go.opentelemetry.io/collector/service.New
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/service/service.go:114
docker-otel-1       | go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/otelcol/collector.go:163
docker-otel-1       | go.opentelemetry.io/collector/otelcol.(*Collector).Run
docker-otel-1       |   go.opentelemetry.io/collector@v0.70.0/otelcol/collector.go:205
docker-otel-1       | main.newCommand.func1
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:122
docker-otel-1       | github.com/spf13/cobra.(*Command).execute
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:916
docker-otel-1       | github.com/spf13/cobra.(*Command).ExecuteC
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:1044
docker-otel-1       | github.com/spf13/cobra.(*Command).Execute
docker-otel-1       |   github.com/spf13/cobra@v1.6.1/command.go:968
docker-otel-1       | main.runInteractive
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:84
docker-otel-1       | main.run
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main_others.go:42
docker-otel-1       | main.main
docker-otel-1       |   github.com/aws-observability/aws-otel-collector/cmd/awscollector/main.go:77
docker-otel-1       | runtime.main
docker-otel-1       |   runtime/proc.go:250
docker-otel-1       | Error: cannot build pipelines: failed to create "awsemf" exporter, in pipeline "metrics": NoAwsRegion: Cannot fetch region variable from config file, environment variables and ec2 metadata.
docker-otel-1       | 2023/02/08 20:36:19 application run finished with error: cannot build pipelines: failed to create "awsemf" exporter, in pipeline "metrics": NoAwsRegion: Cannot fetch region variable from config file, environment variables and ec2 metadata.

[rcdailey]

Ok I think I may see at least one problem. Your example shows that AWS_REGION environment variable must be defined. I assumed it would read the region = us-east-1 value from the config file. The .NET libraries and tooling have done this so far, so I find this unusual.

Another (unrelated) point of confusion is that apparently I can't send logs to cloudwatch from the ADOT collector. Maybe I'm misunderstanding some fundamentals, but I thought one of the main things CloudWatch provided was the ability to aggregate/read logs. At least, I'm able to go there and see logs from ECS containers.

[pingleig]

Normally AWS SDK fetch region from a chain (similar to credential, config, env, EC2 IMDS) but for ADOT, it is using its own 'chain' to combine the region value, ideally it should make the value from ADOT config part of the chain resolvers so the behavior is consistent ... https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/2ce89452c0dad914ef412575e06636c6a4cb402c/internal/aws/awsutil/conn.go#L129-L152

As for log. I haven't worked on ADOT for a while, l remember log in opentelemetry is not GA. ADOT actually does have the ability to write to CloudWatch Logs, but that is only used in awsemfexporter which is sending metrics in the Embed Metrics Format as log. I am not sure about the current work in ADOT, but if you do need agent to send log to cloudwatch log, you can use https://github.com/aws/amazon-cloudwatch-agent The ECS log are sent to CloudWatch should be using the cloudwatch log driver from docker engine https://docs.docker.com/config/containers/logging/awslogs/

[rcdailey]

Thank you for the support. That helps move me along. I was able to get the region working by setting the AWS_REGION environment variable in docker-compose.yml. Also, your advice about AWS EMF was extremely helpful as well.