Downstream AWS service calls annotated as remote

I have just downloaded and tested v2.8.0. I previously had the issue mentioned in issue 64, but that seems to have been resolved. I can see all nodes I expect in the service map now, however the AWS service downstream calls don't look like I would expect.

Normally when you call for example a dynamo DB table, you will see the table name, and type as AWS::DynamoDB::Table. But for dynamodb tables my nodes just have name: "dynamodb.eu-west-1.amazonaws.com" and type: "remote". For SSM calls I just see name: "ssm.eu-west-1.amazonaws.com", type: "remote" etc.

The only node that seems to have the correct annotation besides HTTP calls that actually are remote calls, is the container where the calls originate from, it has the type: "AWS::ECS::Container"

I'm not sure if this is related to the fix or not, but I haven't seen this before.

Hi @dan-lind, I wouldn't think that this would result from the changes I made for 2.8.0, since fixing the bug in #64 was really the only substantive change, but of course you never know. This looks like the higher-level AWS SDK interceptor (which generates the appropriate subsegments for AWS services that you're expecting) is not being picked up and instead the agent is instrumenting the underlying Apache HTTP client that makes the requests to AWS services under the hood.

Couple of questions to start debugging:

Were you seeing these nodes correctly before the 2.8.0 fix, or were you not seeing them at all before (e.g. you never saw them correctly with the agent)?
What version of the AWS SDK are you using?
Are you sure that disco-java-agent-aws-plugin.jar exists in your disco-plugins directory?
If you enable Disco logging as described here, do you see some output before your app starts like the following:

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-aws-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.awsv1.AWSClientInvokeInterceptor, software.amazon.disco.agent.awsv2.AWSClientBuilderInterceptor

I can try to reproduce this issue with the above info.

I had the problem mentioned in #64 when running behind an ALB with 2.7.1, these nodes were missing. When running locally (i.e. not behind an ALB), I saw these nodes using 2.7.1, with correct annotations.

I'm still seeing them correctly when running locally with 2.8.0. So the issue only affects those nodes that I was not seeing at all with 2.7.1, but that are now visible with 2.8.0.

I'm using AWS SDK 2.15.3. The disco-java-agent-aws-plugin.jar is there, and I can see the output you mention when enabling Disco logging.

Ok super interesting that it works locally but not after being deployed behind an ALB. That will make it more difficult to reproduce. What service are you using behind the ALB - EC2, ECS, something else? Are you running a spring boot server or something else? Anything else worth mentioning about the setup?

You are using a newer version of the AWS SDK than the version used to test instrumentation (v2.13.70), so maybe there's been some unexpected breaking changes there.

We are running a Spring boot application (2.3.5) on ECS behind the ALB.

@dan-lind do you think you could post your X-Ray debug logs? These are separate from the Disco debug logs, and you can enable them as described here. Alternatively if you could provide a quick CFN/CDK/terraform setup to repro this issue that would be fantastic, I haven't had the bandwidth to reproduce it myself yet.

I'll look into it after the vacations!

Sorry about the late response to this. This is an example of debug output from the UDPEmitter As you can see everything has the namespace remote.

{
  "name" : "Unlock backend service",
  "id" : "448a33500260ea53",
  "start_time" : 1.611999810919E9,
  "trace_id" : "1-60152a42-cd30e8a61a86ffa1303fb80b",
  "end_time" : 1.611999814021E9,
  "subsegments" : [ {
    "name" : "ssm.eu-west-1.amazonaws.com",
    "id" : "267d3866a4bfd901",
    "start_time" : 1.611999811128E9,
    "end_time" : 1.611999811217E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://ssm.eu-west-1.amazonaws.com/"
      },
      "response" : {
        "content_length" : 213,
        "status" : 200
      }
    }
  }, {
    "name" : "dev-services.collaboration-layer.com",
    "id" : "76fa6f6dc9ec2a45",
    "start_time" : 1.611999811306E9,
    "end_time" : 1.611999811637E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "GET",
        "url" : "https://dev-services.collaboration-layer.com/commonaccess/api/common/accessmanagement/V1/applications/accesssummary/user/dxxjmb/application/4750b7c6-7ace-4ecd-b5da-246187f4bb25"
      },
      "response" : {
        "content_length" : 119,
        "status" : 200
      }
    }
  }, {
    "name" : "ssm.eu-west-1.amazonaws.com",
    "id" : "372775e9baf9ade1",
    "start_time" : 1.611999811793E9,
    "end_time" : 1.611999811852E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://ssm.eu-west-1.amazonaws.com/"
      },
      "response" : {
        "content_length" : 310,
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "0aacf0eb9259ab11",
    "start_time" : 1.611999811866E9,
    "end_time" : 1.611999812002E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/token"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "4d69f246acd0cc58",
    "start_time" : 1.611999812008E9,
    "end_time" : 1.611999812264E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "1ce3f5bb06eabe07",
    "start_time" : 1.611999812324E9,
    "end_time" : 1.611999812351E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "5e2fe4506812af84",
    "start_time" : 1.611999812358E9,
    "end_time" : 1.611999812388E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "Post OCR unlock",
    "id" : "0fc5ab3f8a81e9f2",
    "start_time" : 1.611999812417E9,
    "end_time" : 1.611999812667E9,
    "subsegments" : [ {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "73c795643f6f4c19",
      "start_time" : 1.611999812436E9,
      "end_time" : 1.611999812462E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    }, {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "6a960a61efab79c1",
      "start_time" : 1.611999812472E9,
      "end_time" : 1.611999812502E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    }, {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "6527c384c54028d9",
      "start_time" : 1.611999812533E9,
      "end_time" : 1.611999812564E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    } ]
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "0e9a38037d041906",
    "start_time" : 1.611999812669E9,
    "end_time" : 1.611999812701E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "46a74b5e37faa0e4",
    "start_time" : 1.611999812714E9,
    "end_time" : 1.61199981274E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "73d1b83f3ec1288e",
    "start_time" : 1.611999812751E9,
    "end_time" : 1.61199981278E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "35e73824b2136ff3",
    "start_time" : 1.611999812787E9,
    "end_time" : 1.611999812812E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "3acc2ec138a653cb",
    "start_time" : 1.61199981307E9,
    "end_time" : 1.611999813166E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdts/v1/createtriples"
      },
      "response" : {
        "status" : 201
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "1b7e9166f4e263e0",
    "start_time" : 1.61199981317E9,
    "end_time" : 1.611999813392E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "PUT",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdts/v1/createtriples"
      },
      "response" : {
        "status" : 201
      }
    }
  }, {
    "name" : "dynamodb.eu-west-1.amazonaws.com",
    "id" : "12e2cbcd301ab762",
    "start_time" : 1.611999813561E9,
    "end_time" : 1.611999813612E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://dynamodb.eu-west-1.amazonaws.com/"
      },
      "response" : {
        "content_length" : 62,
        "status" : 200
      }
    }
  }, {
    "name" : "dynamodb.eu-west-1.amazonaws.com",
    "id" : "16ce468c0db938f8",
    "start_time" : 1.611999813963E9,
    "end_time" : 1.611999813969E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://dynamodb.eu-west-1.amazonaws.com/"
      },
      "response" : {
        "content_length" : 2,
        "status" : 200
      }
    }
  } ],
  "http" : {
    "request" : {
      "method" : "POST",
      "client_ip" : "172.17.0.6",
      "url" : "http://dev-colaservices.collaboration-layer.com/unlock/v1/request",
      "user_agent" : "AmazonAPIGateway_4r3z7sfwmg"
    },
    "response" : {
      "status" : 200
    }
  },
  "aws" : {
    "ecs" : {
      "container" : "1c9b4a5b0135",
      "container_id" : "1c9b4a5b0135e613a268bb65248b0a1a94fecc22f231adae92604d6c841f3804"
    },
    "ec2" : {
      "instance_id" : "i-0eee3e40786432bf2",
      "availability_zone" : "eu-west-1b",
      "instance_size" : "t3.medium",
      "ami_id" : "ami-0d3c55e1582bd84fd"
    },
    "xray" : {
      "sdk_version" : "2.8.0",
      "sdk" : "X-Ray for Java",
      "auto_instrumentation" : true
    }
  },
  "origin" : "AWS::ECS::Container",
  "service" : {
    "runtime" : "OpenJDK 64-Bit Server VM",
    "runtime_version" : "11.0.8"
  }
}

Besides there there are loads of

ThreadLocalSegmentContext        : Beginning subsegment named: dynamodb.eu-west-1.amazonaws.com

and subsequent

ThreadLocalSegmentContext        : Ending subsegment named: dynamodb.eu-west-1.amazonaws.com

Please let me know if there is any other debug output in particular that you need.

@dan-lind I've just spent a good amount of time investigating this, and it appears to be a problem in general with the AWS SDK V2 on Spring Boot apps. I thought it might have to do with versioning but testing AWS SDK V2 instrumentation on non-spring-boot apps it works regardless of SDK version. From what I gathered, the Disco plugin that instruments the appropriate AWS SDK V2 classes is being loaded, but it completely fails to actually instrument the targeted method (which is build() on all AWS SDK clients) and wrap it with the needed Disco code. There is definitely something with Spring Boot's classloading mechanism, which really does not play well with the agent, that's causing this. It's just interesting it only affects AWS SDK V2 for some reason.

As a workaround for now, you can manually instrument your AWS SDK V2 clients as laid out in the docs, and that should prevent the remote subsegments from being created. I'll continue investigating at a later time.

Hmmm, that is interesting. But it looks to me like there must be something more too it than just spring boot, because if I run the same spring boot app locally (just started with gradle bootRun and running the xray daemon in a local docker container) the traces are recorded correctly. See below trace, which contains several aws namespaces (DynamoDB, SSM etc).

Could it be related some combination of running ECS/Docker/Spring boot?

{
  "name" : "Unlock backend service",
  "id" : "1b62362d33ff6b0a",
  "start_time" : 1.61269258109E9,
  "trace_id" : "1-601fbc65-f25a89220cf44047393e49ee",
  "end_time" : 1.612692586038E9,
  "subsegments" : [ {
    "name" : "Ssm",
    "id" : "439f14f4b7e6df4f",
    "start_time" : 1.612692581369E9,
    "end_time" : 1.612692581786E9,
    "namespace" : "aws",
    "http" : {
      "response" : {
        "content_length" : 213,
        "status" : 200
      }
    },
    "aws" : {
      "retries" : 0,
      "region" : "eu-west-1",
      "operation" : "GetParameter",
      "request_id" : "acdc49ca-ceeb-4042-bc00-95e6ddd5958c"
    }
  }, {
    "name" : "dev-services.collaboration-layer.com",
    "id" : "111f8e20d242da8a",
    "start_time" : 1.612692581993E9,
    "end_time" : 1.612692582371E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "GET",
        "url" : "https://dev-services.collaboration-layer.com/commonaccess/api/common/accessmanagement/V1/applications/accesssummary/user/dxxjmb/application/4750b7c6-7ace-4ecd-b5da-246187f4bb25"
      },
      "response" : {
        "content_length" : 119,
        "status" : 200
      }
    }
  }, {
    "name" : "Ssm",
    "id" : "3c7c01643492ad41",
    "start_time" : 1.612692582545E9,
    "end_time" : 1.612692582753E9,
    "namespace" : "aws",
    "http" : {
      "response" : {
        "content_length" : 310,
        "status" : 200
      }
    },
    "aws" : {
      "retries" : 0,
      "region" : "eu-west-1",
      "operation" : "GetParameter",
      "request_id" : "2345a283-88c3-49ef-b550-7e1caa4f0a24"
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "475011c2b4ed3a74",
    "start_time" : 1.612692582776E9,
    "end_time" : 1.612692583137E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/token"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "2a014d53ad013554",
    "start_time" : 1.612692583232E9,
    "end_time" : 1.612692583734E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "016c3d52ecc5a5b4",
    "start_time" : 1.612692583828E9,
    "end_time" : 1.612692583918E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "26bcaf8964860875",
    "start_time" : 1.612692583924E9,
    "end_time" : 1.612692584004E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "Post OCR unlock",
    "id" : "3629e578b5f838e5",
    "start_time" : 1.612692584011E9,
    "end_time" : 1.612692584362E9,
    "subsegments" : [ {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "781db21c95b4a521",
      "start_time" : 1.612692584034E9,
      "end_time" : 1.612692584187E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    }, {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "4ccf8fb7dee3ed0e",
      "start_time" : 1.61269258419E9,
      "end_time" : 1.612692584359E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    } ]
  }, {
    "name" : "Post OCR unlock",
    "id" : "0b05beab300cfe6c",
    "start_time" : 1.612692584015E9,
    "end_time" : 1.612692584421E9,
    "subsegments" : [ {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "6f5bf6ca2573f609",
      "start_time" : 1.612692584027E9,
      "end_time" : 1.612692584336E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    }, {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "4b06a229a753b987",
      "start_time" : 1.612692584339E9,
      "end_time" : 1.61269258442E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    } ]
  }, {
    "name" : "Post OCR unlock",
    "id" : "7cf42fb03e43148f",
    "start_time" : 1.612692584015E9,
    "end_time" : 1.612692584333E9,
    "subsegments" : [ {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "0094a735af43d1a5",
      "start_time" : 1.612692584027E9,
      "end_time" : 1.612692584107E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    }, {
      "name" : "api.integration.prod.aws.redacted.com",
      "id" : "5a128d85b990dfc6",
      "start_time" : 1.612692584144E9,
      "end_time" : 1.612692584269E9,
      "namespace" : "remote",
      "http" : {
        "request" : {
          "method" : "POST",
          "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
        },
        "response" : {
          "status" : 200
        }
      }
    } ]
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "177a2134949c9d60",
    "start_time" : 1.612692584422E9,
    "end_time" : 1.612692584508E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "52726c66a16da399",
    "start_time" : 1.612692584511E9,
    "end_time" : 1.612692584589E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "0bd15fbcc4db38e1",
    "start_time" : 1.612692584594E9,
    "end_time" : 1.612692584676E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "23d4b0a320fe3dc4",
    "start_time" : 1.612692584681E9,
    "end_time" : 1.612692584766E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdqs/v1/search"
      },
      "response" : {
        "status" : 200
      }
    }
  }, {
    "name" : "api.integration.prod.aws.redacted.com",
    "id" : "28342d553e399f3b",
    "start_time" : 1.612692584775E9,
    "end_time" : 1.612692585162E9,
    "namespace" : "remote",
    "http" : {
      "request" : {
        "method" : "POST",
        "url" : "https://api.integration.prod.aws.redacted.com/devtest/sdip/sdts/v1/createtriples"
      },
      "response" : {
        "status" : 201
      }
    }
  }, {
    "name" : "DynamoDb",
    "id" : "0e0b226e4cadbf9d",
    "start_time" : 1.612692585327E9,
    "end_time" : 1.612692585693E9,
    "namespace" : "aws",
    "http" : {
      "response" : {
        "content_length" : 2,
        "status" : 200
      }
    },
    "aws" : {
      "retries" : 0,
      "region" : "eu-west-1",
      "operation" : "GetItem",
      "table_name" : "dev-cola-unlock-limit",
      "request_id" : "2D727D33IDVL9JC2G00URKH4URVV4KQNSO5AEMVJF66Q9ASUAAJG"
    }
  }, {
    "name" : "DynamoDb",
    "id" : "77e111b721a22b3b",
    "start_time" : 1.612692585795E9,
    "end_time" : 1.612692585965E9,
    "namespace" : "aws",
    "http" : {
      "response" : {
        "content_length" : 2,
        "status" : 200
      }
    },
    "aws" : {
      "retries" : 0,
      "region" : "eu-west-1",
      "operation" : "PutItem",
      "table_name" : "dev-cola-unlock-limit",
      "request_id" : "T3JAKSC9279CU5GC0K7PD3KBPVVV4KQNSO5AEMVJF66Q9ASUAAJG"
    }
  } ],
  "http" : {
    "request" : {
      "method" : "POST",
      "client_ip" : "0:0:0:0:0:0:0:1",
      "url" : "http://localhost:5000/unlock/v1/request",
      "user_agent" : "PostmanRuntime/7.26.10"
    },
    "response" : {
      "status" : 200
    }
  },
  "aws" : {
    "xray" : {
      "sdk_version" : "2.8.0",
      "sdk" : "X-Ray for Java",
      "auto_instrumentation" : true
    }
  },
  "service" : {
    "runtime" : "OpenJDK 64-Bit Server VM",
    "runtime_version" : "11.0.8"
  }
}

@dan-lind I should have been more specific. It actually relates to the fact that Spring Boot uses a custom LaunchedURLClassLoader to load classes when packaged as a jar. When you run your app with gradle bootRun, Gradle packages your app in a way that uses the SystemClassLoader for all classes in the app. This fact is almost always an implementation detail abstracted from users, but not in this case. If you package your app with gradle build and run it à la production with java -javaagent:disco/disco-java-agent.jar=pluginPath=disco/disco-plugins -jar MyApp.jar you should reproduce the issue.

The problem stems from the fact that the LaunchedURLClassLoader is a sibling of the System class loader, which means the two classloaders are completely unaware of each other. The X-Ray Java Agent used to rely on the assumption that all application classes would live in the system classloader along with the Disco plugins used for instrumentation. We then realized this bizarre behavior with Spring Boot, and refactored most of the agent to remove that assumption. It appears that we have made a miss on AWS SDK v2, and need to address it in the Disco AWS SDK V2 instrumentation. This class needs to be changed to no longer directly reference AWS SDK classes. The Disco maintainers are actively working on this fix and it should be out in the next Disco release.

aws / aws-xray-java-agent

Downstream AWS service calls annotated as remote #72