AWS Kinesis KCL 2.X with X-Ray SegmentNotFoundException

[IDUN-BogdanPi]

Hello,

Is X-Ray compatible with the AWS Kinesis KCL 2.X Java?

I have a consumer, very similar to the one described in the AWS docs.

After adding X-Ray:

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-core</artifactId>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-aws-sdk-v2-instrumentor</artifactId>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-aws-sdk-instrumentor</artifactId>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-aws-sdk-v2</artifactId>
            <version>2.15.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-xray-recorder-sdk-spring</artifactId>
        </dependency>

I get errors about missing segments:

{"@timestamp":"2024-01-25T13:51:55.431966+01:00","level":"ERROR","thread_name":"Thread-2","logger_name":"com.amazonaws.xray.strategy.LogErrorContextMissingStrategy","message":"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'DynamoDb': segment cannot be found."}

{"@timestamp":"2024-01-25T13:51:58.326592+01:00","level":"ERROR","thread_name":"prefetch-cache-shardId-000000000003-0000","logger_name":"com.amazonaws.xray.strategy.LogErrorContextMissingStrategy","message":"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'Kinesis': segment cannot be found."}

{"@timestamp":"2024-01-25T14:17:23.430714+01:00","level":"ERROR","thread_name":"cw-metrics-publisher","logger_name":"com.amazonaws.xray.strategy.LogErrorContextMissingStrategy","message":"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'CloudWatch': segment cannot be found."}

I understand the problem is that Kinesis Consumer is started outside a Servlet context so no parent segment is present but I have not found a way to fix this.

I have tried adding the TracingInterceptor to the individual clients but it doesnt seem to work.

       TracingInterceptor tracingInterceptor = new TracingInterceptor();

        ClientOverrideConfiguration configOverride = ClientOverrideConfiguration
                .builder()
                .addExecutionInterceptor(tracingInterceptor)
                .build();

        KinesisAsyncClientBuilder kinesisAsyncClientBuilder = KinesisAsyncClient.builder()
                .overrideConfiguration(configOverride)
                .region(this.region);
        DynamoDbAsyncClientBuilder dynamoDbAsyncClientBuilder = DynamoDbAsyncClient.builder()
                .overrideConfiguration(configOverride)
                .region(region);
        CloudWatchAsyncClientBuilder cloudWatchAsyncClientBuilder = CloudWatchAsyncClient.builder()
                .overrideConfiguration(configOverride)
                .region(region);

I also tried different ways of modifying the Scheduler to manually start a segment but noting seems to work:

 Segment segment = AWSXRay.beginSegment("KinesisProcessingSegment");

        Scheduler scheduler = new Scheduler(
              ....
        );

        Thread schedulerThread = new Thread(() -> {
            try {
                AWSXRay.setTraceEntity(segment); // Set the segment in the new thread's context
                scheduler.run();

            } finally {
                AWSXRay.endSegment(); // End the segment when the thread is done
            }
        });

This is my X-Ray configuration:

   @Bean
    public AWSXRayRecorder awsXRayRecorder() {
        AWSXRayRecorderBuilder builder = AWSXRayRecorderBuilder.standard().withPlugin(new ECSPlugin());
        AWSXRayRecorder recorder = builder.build();
        AWSXRay.setGlobalRecorder(recorder);
        return recorder;
    }

It was my understanding that when using aws-xray-recorder-sdk-aws-sdk-v2-instrumentor all Java v2 clients should be instrumented with traces automatically.

it is also expected that a queue consumer would run outside of a servlet context.

Am I missing something?

Many thanks for your help and time!

[atshaw43]

You are making an asynchronous AWS SDK call. Unfortunately, X-Ray does not support async calls. That async call can be executed after the endSegment() code is called because it is done in another thread. When it is executed after the endSegment() code, we get a segment not found exception as the subsegment does not have anything with witch to attach.

Would you be able to move over to using Open Telemetry? They have solved this problem.

[IDUN-BogdanPi]

You are making an asynchronous AWS SDK call. Unfortunately, X-Ray does not support async calls. That async call can be executed after the endSegment() code is called because it is done in another thread. When it is executed after the endSegment() code, we get a segment not found exception as the subsegment does not have anything with witch to attach.

Would you be able to move over to using Open Telemetry? They have solved this problem.

Thank you for your reply! Yes, I will look into Open Telemetry. I think the AWS Documentation should be updated to reflect clearly what is and what is not supported with X-Ray.