AWS X-Ray Exporter is not recording error message

gautam-nutalapati commented 2 years ago

Describe the bug I have sample app that produces an exception, which is wrapped in otel custom span.

        Span s = span.startSpan();
        try {
            int i = 1 / 0;
        } catch (Exception e) {
            s.recordException(e);
        }

In AWS XRay, I do not see this error log for respective trace.

Steps to reproduce

Create a custom span, and record an exception.
Use awsxray exporter and check the specific trace data in XRay console I created a sample app that can be used to reproduce this: Link

What did you expect to see? Exception information in trace data. I used Jaeger exporter instead of awsxray, and I was able to see exception information in trace data.

What did you see instead? No exception information in trace data. Trace data: Link

What version did you use? Version: 0.44.0 (aws-otel-collector 0.16.0)

What config did you use?

extensions:
  health_check:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 50
    spike_limit_percentage: 30

exporters:
  awsxray:
    region: "${AWS_REGION}"
service:
  telemetry:
    logs:
      level: "debug"
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch/traces]
      exporters: [awsxray]

Environment OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

Additional context Add any other context about the problem here.

sethAmazon commented 2 years ago

Can you tuen on the debug logs and post them.

gautam-nutalapati commented 2 years ago

Below are the debug logs from aws-otel-collector:


Attaching to opentelemetry-demo-app_aws-otel-collector_1
aws-otel-collector_1                          | 2022/02/11 19:34:33 AWS OTel Collector version: v0.16.0
aws-otel-collector_1                          | 2022/02/11 19:34:33 found no extra config, skip it, err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory
aws-otel-collector_1                          | 2022-02-11T19:34:33.880Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.882Z        debug   awsutil@v0.43.0/conn.go:59      Using proxy address:    {"kind": "exporter", "name": "awsxray", "proxyAddr": ""}
aws-otel-collector_1                          | 2022-02-11T19:34:33.883Z        debug   awsutil@v0.43.0/conn.go:128     Fetch region from commandline/config file       {"kind": "exporter", "name": "awsxray", "region": "us-east-1"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.896Z        debug   awsxrayexporter@v0.43.0/xray_client.go:51       Using Endpoint: %s      {"kind": "exporter", "name": "awsxray", "endpoint": "https://xray.us-east-1.amazonaws.com"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.897Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.898Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.903Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T19:34:33.904Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T19:34:33.904Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.905Z        info    filterprocessor@v0.43.0/filter_processor.go:78  Metric filter configured        {"kind": "processor", "name": "filter/metrics", "include match_type": "", "include expressions": [], "include metric names": [], "include metrics with resource attributes": null, "exclude match_type": "strict", "exclude expressions": [], "exclude metric names": ["http_client_duration"], "exclude metrics with resource attributes": null, "checksMetrics": true, "checkResouces": false}
aws-otel-collector_1                          | 2022-02-11T19:34:33.907Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T19:34:33.908Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T19:34:33.908Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.909Z        info    filterprocessor@v0.43.0/filter_processor.go:78  Metric filter configured        {"kind": "processor", "name": "filter/metrics", "include match_type": "", "include expressions": [], "include metric names": [], "include metrics with resource attributes": null, "exclude match_type": "strict", "exclude expressions": [], "exclude metric names": ["http_client_duration"], "exclude metrics with resource attributes": null, "checksMetrics": true, "checkResouces": false}
aws-otel-collector_1                          | 2022-02-11T19:34:33.910Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T19:34:33.911Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T19:34:33.911Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.911Z        info    builder/receivers_builder.go:226        Receiver was built.     {"kind": "receiver", "name": "otlp", "datatype": "metrics"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.911Z        info    builder/receivers_builder.go:226        Receiver was built.     {"kind": "receiver", "name": "otlp", "datatype": "traces"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.911Z        info    service/service.go:82   Starting extensions...
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "pprof"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    pprofextension@v0.43.0/pprofextension.go:78     Starting net/http/pprof server  {"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":":1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "pprof"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    zpagesextension/zpagesextension.go:40   Register Host's zPages  {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.912Z        info    zpagesextension/zpagesextension.go:53   Starting zPages extension       {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "health_check"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    healthcheckextension@v0.43.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "health_check"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    service/service.go:87   Starting exporters...
aws-otel-collector_1                          | 2022-02-11T19:34:33.913Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.915Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.915Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.915Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.915Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    service/service.go:92   Starting processors...
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    service/service.go:97   Starting receivers...
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    builder/receivers_builder.go:68 Receiver is starting... {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.916Z        info    otlpreceiver/otlp.go:69 Starting GRPC server on endpoint 0.0.0.0:4317   {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.917Z        info    builder/receivers_builder.go:73 Receiver started.       {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.917Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.917Z        info    service/telemetry.go:95 Setting up own telemetry...
aws-otel-collector_1                          | 2022-02-11T19:34:33.918Z        info    service/telemetry.go:115        Serving Prometheus metrics      {"address": ":8888", "level": "basic", "service.instance.id": "52b3ac65-028f-4a21-ad86-802324666a23", "service.version": "latest"}
aws-otel-collector_1                          | 2022-02-11T19:34:33.918Z        info    service/collector.go:229        Starting aws-otel-collector...  {"Version": "v0.16.0", "NumCPU": 2}
aws-otel-collector_1                          | 2022-02-11T19:34:33.919Z        info    service/collector.go:124        Everything is ready. Begin running and processing data.
aws-otel-collector_1                          | 2022-02-11T19:34:34.905Z        debug   memorylimiterprocessor/memorylimiter.go:271     Currently used memory.  {"kind": "processor", "name": "memory_limiter", "cur_mem_mib": 6}
aws-otel-collector_1                          | 2022-02-11T19:35:03.885Z        debug   awsxrayexporter@v0.43.0/awsxray.go:54   TracesExporter  {"kind": "exporter", "name": "awsxray", "type": "awsxray", "name": "awsxray", "#spans": 3}
aws-otel-collector_1                          | 2022-02-11T19:35:03.886Z        debug   awsxrayexporter@v0.43.0/awsxray.go:66   request: {
aws-otel-collector_1                          |   TraceSegmentDocuments: ["{\"name\":\"HelloWorldResource.sayHello\",\"id\":\"28011aa1690a6a06\",\"start_time\":1644608093.8903847,\"trace_id\":\"1-6206ba5d-657feb43b5b16ee5ee423bc5\",\"end_time\":1644608094.4466066,\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"code.function\":\"sayHello\",\"code.namespace\":\"com.example.dropwizard.HelloWorldResource\",\"thread.id\":25,\"thread.name\":\"dw-25 - GET /hello-world?name=error\"}},\"parent_id\":\"143d2bac2aaaf114\",\"type\":\"subsegment\"}\n","{\"name\":\"unknown_service:java\",\"id\":\"143d2bac2aaaf114\",\"start_time\":1644608093.794177,\"trace_id\":\"1-6206ba5d-657feb43b5b16ee5ee423bc5\",\"end_time\":1644608094.5186334,\"http\":{\"request\":{\"method\":\"GET\",\"url\":\"http://web-app-using-aws-otel-collector-and-agent:8080/hello-world?name=error\",\"user_agent\":\"curl/7.61.1\",\"client_ip\":\"172.31.0.7\"},\"response\":{\"status\":200,\"content_length\":0}},\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"http.flavor\":\"1.1\",\"http.response_content_length\":34,\"net.transport\":\"ip_tcp\",\"otel.resource.\\\"service.name\":\"web-app-using-aws-otel\",\"otel.resource.container.id\":\"706936d753eb063542907cd6f056b53cc6fdde0b9564be02543310793a09e43a\",\"otel.resource.host.arch\":\"amd64\",\"otel.resource.host.name\":\"706936d753eb\",\"otel.resource.os.description\":\"Linux 5.10.47-linuxkit\",\"otel.resource.os.type\":\"linux\",\"otel.resource.process.command_line\":\"/opt/java/openjdk:bin:java -javaagent:/otel/aws-opentelemetry-agent.jar\",\"otel.resource.process.executable.path\":\"/opt/java/openjdk:bin:java\",\"otel.resource.process.pid\":1,\"otel.resource.process.runtime.description\":\"Eclipse Foundation OpenJDK 64-Bit Server VM 11.0.12+7\",\"otel.resource.process.runtime.name\":\"OpenJDK Runtime Environment\",\"otel.resource.process.runtime.version\":\"11.0.12+7\",\"otel.resource.service.name\":\"unknown_service:java\",\"otel.resource.service.namespace\":\"dev\\\"\",\"otel.resource.telemetry.auto.version\":\"1.10.1-aws\",\"otel.resource.telemetry.sdk.language\":\"java\",\"otel.resource.telemetry.sdk.name\":\"opentelemetry\",\"otel.resource.telemetry.sdk.version\":\"1.10.1\",\"thread.id\":25,\"thread.name\":\"dw-25\"}}}\n","{\"name\":\"random-sleep\",\"id\":\"3fbc4a7e0c28bfba\",\"start_time\":1644608093.8917391,\"trace_id\":\"1-6206ba5d-657feb43b5b16ee5ee423bc5\",\"end_time\":1644608094.4464407,\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"thread.id\":25,\"thread.name\":\"dw-25 - GET /hello-world?name=error\"}},\"parent_id\":\"28011aa1690a6a06\",\"type\":\"subsegment\"}\n"]
aws-otel-collector_1                          | }       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T19:35:04.116Z        debug   awsxrayexporter@v0.43.0/awsxray.go:73   response: {
aws-otel-collector_1                          |   UnprocessedTraceSegments: []
aws-otel-collector_1                          | }       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T19:35:04.874Z        debug   memorylimiterprocessor/memorylimiter.go:271     Currently used memory.  {"kind": "processor", "name": "memory_limiter", "cur_mem_mib": 7}

~``

sethAmazon commented 2 years ago

Can you add logging exporter please.

gautam-nutalapati commented 2 years ago

Thanks for following up on this. Below are the aws-otel-collector logs with loggingexporter in trace pipeline:

Attaching to opentelemetry-demo-app_aws-otel-collector_1
aws-otel-collector_1                          | 2022/02/11 20:51:57 AWS OTel Collector version: v0.16.0
aws-otel-collector_1                          | 2022/02/11 20:51:57 found no extra config, skip it, err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory
aws-otel-collector_1                          | 2022-02-11T20:51:57.801Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "logging"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.802Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.802Z        debug   awsutil@v0.43.0/conn.go:59      Using proxy address:    {"kind": "exporter", "name": "awsxray", "proxyAddr": ""}
aws-otel-collector_1                          | 2022-02-11T20:51:57.803Z        debug   awsutil@v0.43.0/conn.go:128     Fetch region from commandline/config file       {"kind": "exporter", "name": "awsxray", "region": "us-east-1"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.822Z        debug   awsxrayexporter@v0.43.0/xray_client.go:51       Using Endpoint: %s      {"kind": "exporter", "name": "awsxray", "endpoint": "https://xray.us-east-1.amazonaws.com"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.823Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.825Z        info    builder/exporters_builder.go:255        Exporter was built.     {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.830Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T20:51:57.830Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T20:51:57.832Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.832Z        info    filterprocessor@v0.43.0/filter_processor.go:78  Metric filter configured        {"kind": "processor", "name": "filter/metrics", "include match_type": "", "include expressions": [], "include metric names": [], "include metrics with resource attributes": null, "exclude match_type": "strict", "exclude expressions": [], "exclude metric names": ["http_client_duration"], "exclude metrics with resource attributes": null, "checksMetrics": true, "checkResouces": false}
aws-otel-collector_1                          | 2022-02-11T20:51:57.834Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T20:51:57.835Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T20:51:57.835Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.838Z        info    filterprocessor@v0.43.0/filter_processor.go:78  Metric filter configured        {"kind": "processor", "name": "filter/metrics", "include match_type": "", "include expressions": [], "include metric names": [], "include metrics with resource attributes": null, "exclude match_type": "strict", "exclude expressions": [], "exclude metric names": ["http_client_duration"], "exclude metrics with resource attributes": null, "checksMetrics": true, "checkResouces": false}
aws-otel-collector_1                          | 2022-02-11T20:51:57.843Z        info    memorylimiterprocessor/memorylimiter.go:136     Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "total_memory_mib": 5188, "limit_percentage": 50, "spike_limit_percentage": 30}
aws-otel-collector_1                          | 2022-02-11T20:51:57.843Z        info    memorylimiterprocessor/memorylimiter.go:105     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "limit_mib": 2594, "spike_limit_mib": 1556, "check_interval": 1}
aws-otel-collector_1                          | 2022-02-11T20:51:57.843Z        info    builder/pipelines_builder.go:223        Pipeline was built.     {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.844Z        info    builder/receivers_builder.go:226        Receiver was built.     {"kind": "receiver", "name": "otlp", "datatype": "metrics"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.844Z        info    builder/receivers_builder.go:226        Receiver was built.     {"kind": "receiver", "name": "otlp", "datatype": "traces"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.844Z        info    service/service.go:82   Starting extensions...
aws-otel-collector_1                          | 2022-02-11T20:51:57.844Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "health_check"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.845Z        info    healthcheckextension@v0.43.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "health_check"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "pprof"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    pprofextension@v0.43.0/pprofextension.go:78     Starting net/http/pprof server  {"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":":1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "pprof"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    extensions/extensions.go:38     Extension is starting...        {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.846Z        info    zpagesextension/zpagesextension.go:40   Register Host's zPages  {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    zpagesextension/zpagesextension.go:53   Starting zPages extension       {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    extensions/extensions.go:42     Extension started.      {"kind": "extension", "name": "zpages"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    service/service.go:87   Starting exporters...
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "logging"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "logging"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.847Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.851Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "awsprometheusremotewrite"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.851Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.852Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.852Z        info    builder/exporters_builder.go:40 Exporter is starting... {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.852Z        info    builder/exporters_builder.go:48 Exporter started.       {"kind": "exporter", "name": "prometheus"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.852Z        info    service/service.go:92   Starting processors...
aws-otel-collector_1                          | 2022-02-11T20:51:57.853Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.853Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "traces"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.854Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.854Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "metrics/2"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.855Z        info    builder/pipelines_builder.go:54 Pipeline is starting... {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.857Z        info    builder/pipelines_builder.go:65 Pipeline is started.    {"name": "pipeline", "name": "metrics"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.858Z        info    service/service.go:97   Starting receivers...
aws-otel-collector_1                          | 2022-02-11T20:51:57.858Z        info    builder/receivers_builder.go:68 Receiver is starting... {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.863Z        info    otlpreceiver/otlp.go:69 Starting GRPC server on endpoint 0.0.0.0:4317   {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.863Z        info    builder/receivers_builder.go:73 Receiver started.       {"kind": "receiver", "name": "otlp"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.863Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.864Z        info    service/telemetry.go:95 Setting up own telemetry...
aws-otel-collector_1                          | 2022-02-11T20:51:57.889Z        info    service/telemetry.go:115        Serving Prometheus metrics      {"address": ":8888", "level": "basic", "service.instance.id": "f75bbc4c-c6b2-46a1-a4b6-6be5254cfe64", "service.version": "latest"}
aws-otel-collector_1                          | 2022-02-11T20:51:57.890Z        info    service/collector.go:229        Starting aws-otel-collector...  {"Version": "v0.16.0", "NumCPU": 2}
aws-otel-collector_1                          | 2022-02-11T20:51:57.891Z        info    service/collector.go:124        Everything is ready. Begin running and processing data.
aws-otel-collector_1                          | 2022-02-11T20:51:58.833Z        debug   memorylimiterprocessor/memorylimiter.go:271     Currently used memory.  {"kind": "processor", "name": "memory_limiter", "cur_mem_mib": 6}
aws-otel-collector_1                          | 2022-02-11T20:52:27.823Z        debug   awsxrayexporter@v0.43.0/awsxray.go:54   TracesExporter  {"kind": "exporter", "name": "awsxray", "type": "awsxray", "name": "awsxray", "#spans": 3}
aws-otel-collector_1                          | 2022-02-11T20:52:27.825Z        debug   awsxrayexporter@v0.43.0/awsxray.go:66   request: {
aws-otel-collector_1                          |   TraceSegmentDocuments: ["{\"name\":\"random-sleep\",\"id\":\"2032ebdcecf5b9a6\",\"start_time\":1644612738.1382546,\"trace_id\":\"1-6206cc81-12bdc7d796fea885d739b48b\",\"end_time\":1644612738.6392977,\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"thread.id\":25,\"thread.name\":\"dw-25 - GET /hello-world?name=error\"}},\"parent_id\":\"2c2c8c3ebbfc859f\",\"type\":\"subsegment\"}\n","{\"name\":\"unknown_service:java\",\"id\":\"9b92b65f90153f98\",\"start_time\":1644612737.9838662,\"trace_id\":\"1-6206cc81-12bdc7d796fea885d739b48b\",\"end_time\":1644612738.6916578,\"http\":{\"request\":{\"method\":\"GET\",\"url\":\"http://web-app-using-aws-otel-collector-and-agent:8080/hello-world?name=error\",\"user_agent\":\"curl/7.61.1\",\"client_ip\":\"192.168.16.7\"},\"response\":{\"status\":200,\"content_length\":0}},\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"http.flavor\":\"1.1\",\"http.response_content_length\":34,\"net.transport\":\"ip_tcp\",\"otel.resource.\\\"service.name\":\"web-app-using-aws-otel\",\"otel.resource.container.id\":\"e315a28750b019dbabbcba8a0d8a05aa1d36c6a8a80e8a890fc99dd1cbb495f8\",\"otel.resource.host.arch\":\"amd64\",\"otel.resource.host.name\":\"e315a28750b0\",\"otel.resource.os.description\":\"Linux 5.10.47-linuxkit\",\"otel.resource.os.type\":\"linux\",\"otel.resource.process.command_line\":\"/opt/java/openjdk:bin:java -javaagent:/otel/aws-opentelemetry-agent.jar\",\"otel.resource.process.executable.path\":\"/opt/java/openjdk:bin:java\",\"otel.resource.process.pid\":1,\"otel.resource.process.runtime.description\":\"Eclipse Foundation OpenJDK 64-Bit Server VM 11.0.12+7\",\"otel.resource.process.runtime.name\":\"OpenJDK Runtime Environment\",\"otel.resource.process.runtime.version\":\"11.0.12+7\",\"otel.resource.service.name\":\"unknown_service:java\",\"otel.resource.service.namespace\":\"dev\\\"\",\"otel.resource.telemetry.auto.version\":\"1.10.1-aws\",\"otel.resource.telemetry.sdk.language\":\"java\",\"otel.resource.telemetry.sdk.name\":\"opentelemetry\",\"otel.resource.telemetry.sdk.version\":\"1.10.1\",\"thread.id\":25,\"thread.name\":\"dw-25\"}}}\n","{\"name\":\"HelloWorldResource.sayHello\",\"id\":\"2c2c8c3ebbfc859f\",\"start_time\":1644612738.1362245,\"trace_id\":\"1-6206cc81-12bdc7d796fea885d739b48b\",\"end_time\":1644612738.6394405,\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"xray\":{\"sdk\":\"opentelemetry for java\",\"sdk_version\":\"1.10.1\",\"auto_instrumentation\":true}},\"metadata\":{\"default\":{\"code.function\":\"sayHello\",\"code.namespace\":\"com.example.dropwizard.HelloWorldResource\",\"thread.id\":25,\"thread.name\":\"dw-25 - GET /hello-world?name=error\"}},\"parent_id\":\"9b92b65f90153f98\",\"type\":\"subsegment\"}\n"]
aws-otel-collector_1                          | }       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T20:52:28.095Z        debug   awsxrayexporter@v0.43.0/awsxray.go:73   response: {
aws-otel-collector_1                          |   UnprocessedTraceSegments: []
aws-otel-collector_1                          | }       {"kind": "exporter", "name": "awsxray"}
aws-otel-collector_1                          | 2022-02-11T20:52:28.095Z        INFO    loggingexporter/logging_exporter.go:40  TracesExporter  {"#spans": 3}
aws-otel-collector_1                          | 2022-02-11T20:52:28.096Z        DEBUG   loggingexporter/logging_exporter.go:49  ResourceSpans #0
aws-otel-collector_1                          | Resource SchemaURL: https://opentelemetry.io/schemas/1.8.0
aws-otel-collector_1                          | Resource labels:
aws-otel-collector_1                          |      -> "service.name: STRING(web-app-using-aws-otel)
aws-otel-collector_1                          |      -> container.id: STRING(e315a28750b019dbabbcba8a0d8a05aa1d36c6a8a80e8a890fc99dd1cbb495f8)
aws-otel-collector_1                          |      -> host.arch: STRING(amd64)
aws-otel-collector_1                          |      -> host.name: STRING(e315a28750b0)
aws-otel-collector_1                          |      -> os.description: STRING(Linux 5.10.47-linuxkit)
aws-otel-collector_1                          |      -> os.type: STRING(linux)
aws-otel-collector_1                          |      -> process.command_line: STRING(/opt/java/openjdk:bin:java -javaagent:/otel/aws-opentelemetry-agent.jar)
aws-otel-collector_1                          |      -> process.executable.path: STRING(/opt/java/openjdk:bin:java)
aws-otel-collector_1                          |      -> process.pid: INT(1)
aws-otel-collector_1                          |      -> process.runtime.description: STRING(Eclipse Foundation OpenJDK 64-Bit Server VM 11.0.12+7)
aws-otel-collector_1                          |      -> process.runtime.name: STRING(OpenJDK Runtime Environment)
aws-otel-collector_1                          |      -> process.runtime.version: STRING(11.0.12+7)
aws-otel-collector_1                          |      -> service.name: STRING(unknown_service:java)
aws-otel-collector_1                          |      -> service.namespace: STRING(dev")
aws-otel-collector_1                          |      -> telemetry.auto.version: STRING(1.10.1-aws)
aws-otel-collector_1                          |      -> telemetry.sdk.language: STRING(java)
aws-otel-collector_1                          |      -> telemetry.sdk.name: STRING(opentelemetry)
aws-otel-collector_1                          |      -> telemetry.sdk.version: STRING(1.10.1)
aws-otel-collector_1                          | InstrumentationLibrarySpans #0
aws-otel-collector_1                          | InstrumentationLibraryMetrics SchemaURL: 
aws-otel-collector_1                          | InstrumentationLibrary somename 1.0
aws-otel-collector_1                          | Span #0
aws-otel-collector_1                          |     Trace ID       : 6206cc8112bdc7d796fea885d739b48b
aws-otel-collector_1                          |     Parent ID      : 2c2c8c3ebbfc859f
aws-otel-collector_1                          |     ID             : 2032ebdcecf5b9a6
aws-otel-collector_1                          |     Name           : random-sleep
aws-otel-collector_1                          |     Kind           : SPAN_KIND_INTERNAL
aws-otel-collector_1                          |     Start time     : 2022-02-11 20:52:18.1382545 +0000 UTC
aws-otel-collector_1                          |     End time       : 2022-02-11 20:52:18.6392978 +0000 UTC
aws-otel-collector_1                          |     Status code    : STATUS_CODE_UNSET
aws-otel-collector_1                          |     Status message : 
aws-otel-collector_1                          | Attributes:
aws-otel-collector_1                          |      -> thread.name: STRING(dw-25 - GET /hello-world?name=error)
aws-otel-collector_1                          |      -> thread.id: INT(25)
aws-otel-collector_1                          | Events:
aws-otel-collector_1                          | SpanEvent #0
aws-otel-collector_1                          |      -> Name: exception
aws-otel-collector_1                          |      -> Timestamp: 2022-02-11 20:52:18.6259161 +0000 UTC
aws-otel-collector_1                          |      -> DroppedAttributesCount: 0
aws-otel-collector_1                          |      -> Attributes:
aws-otel-collector_1                          |          -> exception.message: STRING(/ by zero)
aws-otel-collector_1                          |          -> exception.stacktrace: STRING(java.lang.ArithmeticException: / by zero
aws-otel-collector_1                          |         at com.example.dropwizard.HelloWorldResource.generateError(HelloWorldResource.java:110)
aws-otel-collector_1                          |         at com.example.dropwizard.HelloWorldResource.sayHello(HelloWorldResource.java:74)
aws-otel-collector_1                          |         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
aws-otel-collector_1                          |         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
aws-otel-collector_1                          |         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
aws-otel-collector_1                          |         at java.base/java.lang.reflect.Method.invoke(Unknown Source)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
aws-otel-collector_1                          |         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
aws-otel-collector_1                          |         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
aws-otel-collector_1                          |         at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
aws-otel-collector_1                          |         at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
aws-otel-collector_1                          |         at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
aws-otel-collector_1                          |         at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
aws-otel-collector_1                          |         at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
aws-otel-collector_1                          |         at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
aws-otel-collector_1                          |         at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
aws-otel-collector_1                          |         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
aws-otel-collector_1                          |         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
aws-otel-collector_1                          |         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
aws-otel-collector_1                          |         at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
aws-otel-collector_1                          |         at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
aws-otel-collector_1                          |         at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
aws-otel-collector_1                          |         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
aws-otel-collector_1                          |         at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313)
aws-otel-collector_1                          |         at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.Server.handle(Server.java:516)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
aws-otel-collector_1                          |         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
aws-otel-collector_1                          |         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
aws-otel-collector_1                          |         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
aws-otel-collector_1                          |         at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
aws-otel-collector_1                          |         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
aws-otel-collector_1                          |         at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
aws-otel-collector_1                          |         at java.base/java.lang.Thread.run(Unknown Source)
aws-otel-collector_1                          | )
aws-otel-collector_1                          |          -> exception.type: STRING(java.lang.ArithmeticException)
aws-otel-collector_1                          | InstrumentationLibrarySpans #1
aws-otel-collector_1                          | InstrumentationLibraryMetrics SchemaURL: 
aws-otel-collector_1                          | InstrumentationLibrary io.opentelemetry.jetty-8.0 1.10.1-aws
aws-otel-collector_1                          | Span #0
aws-otel-collector_1                          |     Trace ID       : 6206cc8112bdc7d796fea885d739b48b
aws-otel-collector_1                          |     Parent ID      : 
aws-otel-collector_1                          |     ID             : 9b92b65f90153f98
aws-otel-collector_1                          |     Name           : /hello-world
aws-otel-collector_1                          |     Kind           : SPAN_KIND_SERVER
aws-otel-collector_1                          |     Start time     : 2022-02-11 20:52:17.983866 +0000 UTC
aws-otel-collector_1                          |     End time       : 2022-02-11 20:52:18.6916578 +0000 UTC
aws-otel-collector_1                          |     Status code    : STATUS_CODE_UNSET
aws-otel-collector_1                          |     Status message : 
aws-otel-collector_1                          | Attributes:
aws-otel-collector_1                          |      -> http.flavor: STRING(1.1)
aws-otel-collector_1                          |      -> http.target: STRING(/hello-world?name=error)
aws-otel-collector_1                          |      -> net.peer.port: INT(38818)
aws-otel-collector_1                          |      -> net.transport: STRING(ip_tcp)
aws-otel-collector_1                          |      -> http.host: STRING(web-app-using-aws-otel-collector-and-agent:8080)
aws-otel-collector_1                          |      -> net.peer.ip: STRING(192.168.16.7)
aws-otel-collector_1                          |      -> thread.name: STRING(dw-25)
aws-otel-collector_1                          |      -> http.user_agent: STRING(curl/7.61.1)
aws-otel-collector_1                          |      -> http.status_code: INT(200)
aws-otel-collector_1                          |      -> http.server_name: STRING(web-app-using-aws-otel-collector-and-agent)
aws-otel-collector_1                          |      -> http.response_content_length: INT(34)
aws-otel-collector_1                          |      -> http.scheme: STRING(http)
aws-otel-collector_1                          |      -> http.method: STRING(GET)
aws-otel-collector_1                          |      -> thread.id: INT(25)
aws-otel-collector_1                          | InstrumentationLibrarySpans #2
aws-otel-collector_1                          | InstrumentationLibraryMetrics SchemaURL: 
aws-otel-collector_1                          | InstrumentationLibrary io.opentelemetry.jaxrs-2.0-common 1.10.1-aws
aws-otel-collector_1                          | Span #0
aws-otel-collector_1                          |     Trace ID       : 6206cc8112bdc7d796fea885d739b48b
aws-otel-collector_1                          |     Parent ID      : 9b92b65f90153f98
aws-otel-collector_1                          |     ID             : 2c2c8c3ebbfc859f
aws-otel-collector_1                          |     Name           : HelloWorldResource.sayHello
aws-otel-collector_1                          |     Kind           : SPAN_KIND_INTERNAL
aws-otel-collector_1                          |     Start time     : 2022-02-11 20:52:18.1362246 +0000 UTC
aws-otel-collector_1                          |     End time       : 2022-02-11 20:52:18.6394407 +0000 UTC
aws-otel-collector_1                          |     Status code    : STATUS_CODE_UNSET
aws-otel-collector_1                          |     Status message : 
aws-otel-collector_1                          | Attributes:
aws-otel-collector_1                          |      -> code.function: STRING(sayHello)
aws-otel-collector_1                          |      -> code.namespace: STRING(com.example.dropwizard.HelloWorldResource)
aws-otel-collector_1                          |      -> thread.name: STRING(dw-25 - GET /hello-world?name=error)
aws-otel-collector_1                          |      -> thread.id: INT(25)
aws-otel-collector_1                          | 
aws-otel-collector_1                          | 2022-02-11T20:52:28.799Z        debug   memorylimiterprocessor/memorylimiter.go:271     Currently used memory.  {"kind": "processor", "name": "memory_limiter", "cur_mem_mib": 10}

sethAmazon commented 2 years ago

Looking at the trace segment request it does not have the exception

sethAmazon commented 2 years ago

It looks like the status code is STATUS_CODE_UNSET which will not add an exception. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awsxrayexporter/internal/translator/cause.go#L35-L40 I'm not sure if the issue is in the receiver or in the sdk. @willarmiros can you please take a look.

BT-22 commented 2 years ago

Hello, is there an update for this issue?

While in the code mentioned above it is clear that in order to get the stack trace, the span has to be set to error status, I think in a way it probably defeats the purpose of recordException if we are not getting the stack trace.

But, why would one record an exception but not wish to set the span in error? One use case is in one's retry logic, we may want to record an exception with stack trace to understand what has caused the retry; however, if the retry works most time, we may not want to get the error state triggered alerts.

willarmiros commented 2 years ago

Hi @BT-22, sorry for the delayed response and thank you for posting this. We are aware of this issue and will be addressing it as soon as we can.

willarmiros commented 2 years ago

I believe this is resolved by #12643, let's keep this open til someone can verify the fix with the latest collector version

BT-22 commented 2 years ago

Hi @willarmiros Just to make sure I understand the release schedule correctly - The latest ADOT Collector release is v0.20.0 and this fix is not there yet; we'll need v0.21.0 perhaps to test with an official release?

willarmiros commented 2 years ago

Yes that's right, sorry for being unclear. The next (unreleased as of yet) ADOT Collector release should include this fix.

BT-22 commented 2 years ago

Hi @willarmiros Just to mention that I verified the fix with the v0.21.1 AWS OTel Collector. I recorded an exception without setting the span in error status, and the trace includes the full stack trace.

Not sure if https://aws-otel.github.io/docs/getting-started/x-ray#configuring-the-aws-x-ray-exporter can use some adjustments, because after the fix, span status no longer matters in term of capturing stack trace. (Although the doc is not very explicit in saying when the stack trace won't be captured)

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

github-actions[bot] commented 1 year ago

This issue has been closed as inactive because it has been stale for 120 days with no activity.

open-telemetry / opentelemetry-collector-contrib

AWS X-Ray Exporter is not recording error message #7781