apache / pekko

Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala
https://pekko.apache.org/
Apache License 2.0
1.18k stars 142 forks source link

With entity passivation enabled the shard region will cause a lot of unhandled ShardsUpdated during shutdown #1341

Closed nvollmar closed 3 months ago

nvollmar commented 4 months ago

We encounter hundreds of these during a node shutdown

unhandled message from Actor[pekko://main/system/sharding/transactionsAggregates#-1585167465] to Actor[pekko://main/system/sharding/transactionsAggregates/4#-516417762]: ShardsUpdated(5)

Root cause is to link to link ShardRegion sending updates for each terminated shard when passivation strategy is used:

      if (settings.passivationStrategy != ClusterShardingSettings.NoPassivationStrategy) {
        shards.values.foreach(_ ! ShardsUpdated(shards.size))
      }

See here

That message is not handled/ignored by the Shard during termination:

       context.become {
          case Terminated(ref) => receiveTerminated(ref)
        }

See here

There would be two ways to fix this:

  1. At this time the ShardRegion property gracefulShutdownInProgress is already set to true - check for that in the if and don't send any more updates
  2. During termination in the Shard match and ignore the ShardsUpdated