com.amazonaws.AbortedException in KinesisShardGetter

[geota]

We are periodically seeing these messages in our logs after our topology has been running for a number of hours. They are crashing the spout. We have forked the code hosted here to add support for pushing failed messages to a SQS Dead Letter Queue. But we don't think that is related. This issue is meant as a discussion, we will update this issue as we make progress finding the root cause.. Any tips are appreciated.

• 10 Shard Kinesis
• 10 parallelism on the Kinesis Spout

We have seen this issue across two of our topologies so we don't think its related to any of our business logic.

{
    "timestamp": "2015-06-25T17:36:22.375+0000",
    "level": "ERROR",
    "thread": "Thread-21-kinesis-spout",
    "logger": "com.amazonaws.services.kinesis.stormspout.KinesisShardGetter",
    "message": "KinesisShardGetter[shardId=shardId-000000000006] Caught exception when fetching records for shardId-000000000006",
    "context": "default",
    "exception": "com.amazonaws.AbortedException: \n\tat com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:51)\n\tat com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:71)\n\tat com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)\n\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore(UTF8StreamJsonParser.java:180)\n\tat com.fasterxml.jackson.core.base.ParserBase.loadMoreGuaranteed(ParserBase.java:460)\n\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2306)\n\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287)\n\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286)\n\tat com.amazonaws.transform.JsonUnmarshallerContextImpl.readCurrentJsonTokenValue(JsonUnmarshallerContextImpl.java:129)\n\tat com.amazonaws.transform.JsonUnmarshallerContextImpl.readText(JsonUnmarshallerContextImpl.java:123)\n\tat com.amazonaws.transform.SimpleTypeJsonUnmarshallers$ByteBufferJsonUnmarshaller.unmarshall(SimpleTypeJsonUnmarshallers.java:185)\n\tat com.amazonaws.services.kinesis.model.transform.RecordJsonUnmarshaller.unmarshall(RecordJsonUnmarshaller.java:54)\n\tat com.amazonaws.services.kinesis.model.transform.RecordJsonUnmarshaller.unmarshall(RecordJsonUnmarshaller.java:31)\n\tat com.amazonaws.transform.ListUnmarshaller.unmarshallJsonToList(ListUnmarshaller.java:93)\n\tat com.amazonaws.transform.ListUnmarshaller.unmarshall(ListUnmarshaller.java:43)\n\tat com.amazonaws.services.kinesis.model.transform.GetRecordsResultJsonUnmarshaller.unmarshall(GetRecordsResultJsonUnmarshaller.java:50)\n\tat com.amazonaws.services.kinesis.model.transform.GetRecordsResultJsonUnmarshaller.unmarshall(GetRecordsResultJsonUnmarshaller.java:31)\n\tat com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:106)\n\tat com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:42)\n\tat com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:975)\n\tat com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:702)\n\tat com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:461)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:296)\n\tat com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2498)\n\tat com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1133)\n\tat com.amazonaws.services.kinesis.stormspout.KinesisShardGetter.safeGetRecords(KinesisShardGetter.java:175)\n\tat com.amazonaws.services.kinesis.stormspout.KinesisShardGetter.getNext(KinesisShardGetter.java:74)\n\tat com.amazonaws.services.kinesis.stormspout.BufferedGetter.rebuffer(BufferedGetter.java:126)\n\tat com.amazonaws.services.kinesis.stormspout.BufferedGetter.ensureBuffered(BufferedGetter.java:119)\n\tat com.amazonaws.services.kinesis.stormspout.BufferedGetter.getNext(BufferedGetter.java:71)\n\tat com.amazonaws.services.kinesis.stormspout.KinesisSpout.nextTuple(KinesisSpout.java:181)\n\tat backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565)\n\tat backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)\n\tat clojure.lang.AFn.run(AFn.java:24)\n\tat java.lang.Thread.run(Thread.java:745)\n"
}

[mrdlmndr]

I am also facing the same problem. Hi Geota, have you solved this problem?

[rahulashok]

I am having the same issue as well. Unable to figure out a solution...

[geota]

wrong issue, created #20 - we have not seen this occur for hte last few weeks. closing this out