search

akka / akka-persistence-dynamodb

DynamoDB plugin for Akka Persistence
https://doc.akka.io/libraries/akka-persistence-dynamodb/current/
Other
2 stars 2 forks source link

Frequent fatal rejected envelopes from backtracking while recovering large backlog #95

Open leviramsey opened 2 days ago

leviramsey commented 2 days ago

We just started a DynamoDB projection from a DynamoDB journal (using the 24.10.1 BOM). The journal has been being written to heavily for several days, so the projection has a backlog to start of a few million events.

We are observing consistent failures and restarts of the projection with

Failed to process envelopes with offsets from [TimestampOffset(2024-11-08T22:25:45.722977Z, 2024-11-13T17:55:53.021702Z, Map(EntityType|someId0 -> 9))] to [TimestampOffset(2024-11-08T22:25:04.180012Z, 2024-11-13T17:55:53.022282Z, Map(EntityType|someId1 -> 5))] after [4] attempts. Projection will stop as defined by recovery strategy. MDC:[akkaAddress=akka://[!FILTERED!]@10.236.134.254:2551, akkaUid=-7361407520859643022, sourceThread=actorSystemName-akka.actor.default-dispatcher-30, akkaSource=DynamoDBProjectionImpl(akka://actorSystemName), sourceActorSystem=actorSystemName, akkaTimestamp=17:56:05.236UTC] akka.projection.dynamodb.internal.DynamoDBOffsetStore$RejectedEnvelope: Rejected envelope from backtracking, persistenceId [EntityType|someId3], seqNr [24] due to unexpected sequence number.

These restarts are resulting in rerunning queries resulting in wasted RCU consumption.

leviramsey commented 2 days ago

Possibly related to #70 and #79