akkadotnet / akka.net

Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
http://getakka.net
Other
4.72k stars 1.04k forks source link

Akka.Cluster.Sharding: message `Persistent` was not handled by the coordinator #6133

Open Aaronontheweb opened 2 years ago

Aaronontheweb commented 2 years ago

Version Information Version of Akka.NET? v1.5.0-alpha1 Which Akka.NET Modules? Akka.Cluster.Sharding

Describe the bug

It looks like the Akka.Cluster.Sharding PersistentCoordinator is not able to correctly run its shard migration, as we see error messages like this at startup:

[INFO][10/4/2022 5:01:34 PM][Thread 0047][akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator] Message [Persistent] to [akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator#8081
55823] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO][10/4/2022 5:01:34 PM][Thread 0047][akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator] Message [Persistent] to [akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator#8081
55823] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO][10/4/2022 5:01:34 PM][Thread 0047][akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator] Message [Persistent] to [akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator#8081
55823] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO][10/4/2022 5:01:34 PM][Thread 0047][akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator] Message [Persistent] to [akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator#8081
55823] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO][10/4/2022 5:01:34 PM][Thread 0047][akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator] Message [Persistent] to [akka://SqlSharding/system/sharding/productsCoordinator/singleton/coordinator#8081
55823] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [00:05:00]. This logging can be turned off or adjusted with con

And in addition to that, the Akka.Persistence.SqlServer journal returns no results for SM - indicating that the migration of previous remember-entities data was executed:

/****** Script for SelectTopNRows command from SSMS  ******/
SELECT TOP (1000) [Ordering]
      ,[PersistenceId]
      ,[SequenceNr]
      ,[Timestamp]
      ,[IsDeleted]
      ,[Manifest]
      ,[Payload]
      ,[Tags]
      ,[SerializerId]
  FROM [Akka].[dbo].[ShardingEventJournal]
  WHERE Manifest = 'SM'

To Reproduce

Reproduction instructions and data available here: https://github.com/petabridge/akkadotnet-code-samples/pull/112

Expected behavior

All remembered-entities should start automatically.

Actual behavior

None do.

Additional context

I used a pbm actor hierarchy -s akka://SqlSharding/ -d 6 command to do a dump of the entire actor hierarchy after the sharding system started correctly and here's what I found:

/                                                                                                        
//system                                                                                                 
//system/akka.persistence.journal.sharding                                                               
//system/akka.persistence.snapshot-store.sharding                                                        
//system/cluster                                                                                         
//system/cluster/core                                                                                    
//system/cluster/core/daemon                                                                             
//system/cluster/core/daemon/heartbeatSender                                                             
//system/cluster/core/publisher                                                                          
//system/cluster/heartbeatReceiver                                                                       
//system/clusterEventBusListener                                                                         
//system/deadLetterListener                                                                              
//system/endpointManager                                                                                 
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1        
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1/endpoin
Writer                                                                                                   
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1/endpoin
Writer/endpointReader-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1                                  
//system/EventStreamUnsubscriber-1                                                                       
//system/IO-TCP                                                                                          
//system/IO-TCP/$a                                                                                       
//system/IO-TCP/$a/$a                                                                                    
//system/log1-DefaultLogger                                                                              
//system/recoveryPermitter                                                                               
//system/remote-deployment-watcher                                                                       
//system/remote-watcher                                                                                  
//system/remoting-terminator                                                                             
//system/sharding                                                                                        
//system/sharding/products                                                                               
//system/sharding/productsCoordinator                                                                    
//system/sharding/productsCoordinator/$a                                                                 
//system/sharding/productsCoordinator/singleton                                                          
//system/sharding/productsCoordinator/singleton/coordinator                                              
//system/transports                                                                                      
//system/transports/akkaprotocolmanager.tcp.0                                                            
//system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1  
//user                                                                                                   
//user/$a                                                                                                
//user/pbm-local-clients                                                                                 
//user/pbm-uptime                                                                                        
//user/petabridge.cmd                                                                                    
//user/petabridge.cmd/127.0.0.1%3A59995                                                                  
//user/petabridge.cmd/127.0.0.1%3A59995/handler                                                          
//user/petabridge.cmd/127.0.0.1%3A59995/handler/actor                                                    
//user/petabridge.cmd/127.0.0.1%3A59995/handler/actor/handler-0                                          
//user/petabridge.cmd/127.0.0.1%3A59995/handler/actor/handler-0/$a                                       
//user/petabridge.cmd-log-memorizer                                                                      
//user/product-index                                                                                     
//user/product-index/$a                                                                                  
//user/product-index/product-index                                                                       
//user/product-proxy                                                                                     

This verifies that none of the remembered entities or shards started correctly.

Aaronontheweb commented 2 years ago

HOCON configuration for this reproduction:

akka{
    cluster.sharding{
        # use separate journal implementations for Akka.Cluster.Sharding
        journal-plugin-id = akka.persistence.journal.sharding
        snapshot-plugin-id = akka.persistence.snapshot-store.sharding

        state-store-mode = persistence
        remember-entities-store = eventsourced
        remember-entities = on
        passivate-idle-entity-after = off

        fail-on-invalid-entity-state-transition = on
    }

    persistence{
        journal{
            sharding{
                # qualified type name of the SQL Server persistence journal actor
                class = "Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer"

                # connection string used for database access
                connection-string = ""

                # SQL server table corresponding with persistent journal
                table-name = ShardingEventJournal
                metadata-table-name = ShardingMetadata

                # should corresponding journal table be initialized automatically
                auto-initialize = on

                event-adapters {        
                    tagger = "SqlSharding.Shared.MessageTagger, SqlSharding.Shared"
                    coordinator-migration = "Akka.Cluster.Sharding.OldCoordinatorStateMigrationEventAdapter, Akka.Cluster.Sharding"
                }

                event-adapter-bindings = {    
                    "System.Object" = tagger
                    "Akka.Cluster.Sharding.ShardCoordinator+IDomainEvent, Akka.Cluster.Sharding" = coordinator-migration
                }
            }
        }

        snapshot-store{
            sharding{
                # qualified type name of the SQL Server persistence journal actor
                class = "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"

                # connection string used for database access
                connection-string = ""

                # SQL server table corresponding with persistent journal
                table-name = ShardingSnapshotStore

                # should corresponding journal table be initialized automatically
                auto-initialize = on
            }           
        }
    }
}

This is supposed to be stage 1 of the Akka.NET v1.5 migration.

Aaronontheweb commented 2 years ago

In case it matters: both #6132 and this issue were reproduced using Akka.Persistence.SqlServer.

Aaronontheweb commented 2 years ago

Changing the sharding configuration to read:

akka.cluster.sharding{
         state-store-mode = ddata
        remember-entities-store = eventsourced
}

Worked just fine and successfully instantiated all remembered entities and shards, but it still produced #6132

Can verify that the entities were all started via pbm:

λ  pbm actor hierarchy -s akka://SqlSharding/ -d 6
/
//system
//system/akka.persistence.journal.sharding
//system/akka.persistence.journal.sql-server
//system/akka.persistence.snapshot-store.sharding
//system/akka.persistence.snapshot-store.sql-server
//system/cluster
//system/cluster/core
//system/cluster/core/daemon
//system/cluster/core/daemon/heartbeatSender
//system/cluster/core/publisher
//system/cluster/heartbeatReceiver
//system/clusterEventBusListener
//system/deadLetterListener
//system/endpointManager
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1/endpointWriter
//system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1
//system/EventStreamUnsubscriber-1
//system/IO-TCP
//system/IO-TCP/$a
//system/IO-TCP/$a/$a
//system/log1-DefaultLogger
//system/recoveryPermitter
//system/remote-deployment-watcher
//system/remote-watcher
//system/remoting-terminator
//system/sharding
//system/sharding/hostReplicator
//system/sharding/hostReplicator/durableStore
//system/sharding/products
//system/sharding/products/0
//system/sharding/products/0/791f0f68-5f57-4be4-9e08-24e66fe89e4e
//system/sharding/products/0/RememberEntitiesStore
//system/sharding/products/10
//system/sharding/products/10/7a4a4fb4-85ce-41a1-a89d-015c41bd2e0f
//system/sharding/products/10/9e33a032-ce18-4922-85f7-6abfa0d23d6a
//system/sharding/products/10/RememberEntitiesStore
//system/sharding/products/11
//system/sharding/products/11/bfbdd6aa-7942-488f-8d74-cbf43b99d55f
//system/sharding/products/11/RememberEntitiesStore
//system/sharding/products/12
//system/sharding/products/12/2b16d499-f41c-4196-a7a3-78a148153158
//system/sharding/products/12/RememberEntitiesStore
//system/sharding/products/14
//system/sharding/products/14/402d3db2-2145-4e4a-b094-f1bb06c0d984
//system/sharding/products/14/7226945c-61a8-47e7-9c5a-c170187d6504
//system/sharding/products/14/c6f10055-6eae-4999-9d7b-1efd79c5bcf4
//system/sharding/products/14/RememberEntitiesStore
//system/sharding/products/16
//system/sharding/products/16/263ab2dc-7106-4023-8bdb-887019af5c09
//system/sharding/products/16/567bc4d9-19f9-463d-be72-7b396ed913cb
//system/sharding/products/16/RememberEntitiesStore
//system/sharding/products/18
//system/sharding/products/18/5040b243-7d97-4ccb-b789-d8fae36f16d1
//system/sharding/products/18/a6f0a8bd-1187-4863-be15-27556fbb79de
//system/sharding/products/18/fe0731aa-b39b-4769-9805-4abe8bd3b9b3
//system/sharding/products/18/RememberEntitiesStore
//system/sharding/products/20
//system/sharding/products/20/b8662e65-be6b-4cfa-9b72-0ccab323552d
//system/sharding/products/20/cf5d0c7b-829e-4695-8c7c-2f96a15180fa
//system/sharding/products/20/RememberEntitiesStore
//system/sharding/products/22
//system/sharding/products/22/78f210f1-7658-4714-a971-f6bce6a7c342
//system/sharding/products/22/f83d5a6d-486d-4304-967c-757b0b4a93d8
//system/sharding/products/22/RememberEntitiesStore
//system/sharding/products/24
//system/sharding/products/24/cd1ae778-254c-4fdd-91da-4394bf6cb032
//system/sharding/products/24/RememberEntitiesStore
//system/sharding/products/25
//system/sharding/products/25/3c7ec36c-e9fd-45b2-bc33-060232f2cadd
//system/sharding/products/25/3f5c54d3-c559-410d-b500-435424210194
//system/sharding/products/25/46eca73b-319e-42e0-9c89-50a3b33b82e9
//system/sharding/products/25/RememberEntitiesStore
//system/sharding/products/28
//system/sharding/products/28/4bc6dc56-b1ff-4187-a48d-24adb9b23de5
//system/sharding/products/28/RememberEntitiesStore
//system/sharding/products/29
//system/sharding/products/29/f4eb1f6f-b692-4b05-aff6-f671d6b3419f
//system/sharding/products/29/RememberEntitiesStore
//system/sharding/products/3
//system/sharding/products/3/27dc2b45-cc35-48e7-baad-806c554797c1
//system/sharding/products/3/656bd076-bf7d-4fbb-ae5f-1a245598a9b5
//system/sharding/products/3/RememberEntitiesStore
//system/sharding/products/33
//system/sharding/products/33/51eec4bc-c34b-40ad-acfd-00f7aae80fad
//system/sharding/products/33/5d18fb53-3966-4250-a264-8eb286e9a4e0
//system/sharding/products/33/RememberEntitiesStore
//system/sharding/products/35
//system/sharding/products/35/2d7ad2fa-dd02-47d4-8da8-def8a7cb4b4b
//system/sharding/products/35/8f5d2627-6fde-41ab-9c47-f9285d2e3087
//system/sharding/products/35/RememberEntitiesStore
//system/sharding/products/36
//system/sharding/products/36/536cb1c5-7734-4503-9870-22e2352d7d51
//system/sharding/products/36/RememberEntitiesStore
//system/sharding/products/38
//system/sharding/products/38/3607c89a-e71e-41c4-aee9-a1c665767bf4
//system/sharding/products/38/8381f84d-9f7e-4963-90de-11df8f4ddb50
//system/sharding/products/38/RememberEntitiesStore
//system/sharding/products/39
//system/sharding/products/39/561f10f8-ee9b-4798-9248-99b2a3971285
//system/sharding/products/39/RememberEntitiesStore
//system/sharding/products/41
//system/sharding/products/41/172bc0dd-bcdf-4109-85b5-525b0fb7a01e
//system/sharding/products/41/9bfa83d4-64aa-4367-a747-9ba3b107ef20
//system/sharding/products/41/9e6133cc-b619-4ca5-9166-7f84f03dba52
//system/sharding/products/41/RememberEntitiesStore
//system/sharding/products/45
//system/sharding/products/45/08dd424f-ed76-424a-88ed-5c70c720a6b6
//system/sharding/products/45/RememberEntitiesStore
//system/sharding/products/46
//system/sharding/products/46/ac68d893-cf23-40d5-8113-f97a9b7c9193
//system/sharding/products/46/RememberEntitiesStore
//system/sharding/products/48
//system/sharding/products/48/0f151372-581f-4f7f-bac1-d1dffdd3715e
//system/sharding/products/48/244d185e-92a6-4cad-84da-a4c24a125e2b
//system/sharding/products/48/RememberEntitiesStore
//system/sharding/products/49
//system/sharding/products/49/45524a7d-cb70-4e32-9bd3-e58afa6fb129
//system/sharding/products/49/7bd6e6b0-3d31-4ad9-a4ff-547cfd712aab
//system/sharding/products/49/d512cbc1-e2d2-40c3-a100-9ed8a3728d22
//system/sharding/products/49/RememberEntitiesStore
//system/sharding/products/5
//system/sharding/products/5/242f25b6-15c6-4671-bbd3-91e3ea02ab8d
//system/sharding/products/5/52d54b16-b2f8-4503-a6cd-88f2e3fa29c0
//system/sharding/products/5/c1d62e6c-9662-41a2-9fed-f17ae8c3dd2f
//system/sharding/products/5/RememberEntitiesStore
//system/sharding/products/6
//system/sharding/products/6/7f9d320d-4d6a-47f9-9df7-c84d5a2658e2
//system/sharding/products/6/RememberEntitiesStore
//system/sharding/products/7
//system/sharding/products/7/61eb3e6c-7a44-4ef3-8ddf-9adfa341f0b4
//system/sharding/products/7/63bfdc42-3c19-45b0-9e5e-cfa966eb732f
//system/sharding/products/7/RememberEntitiesStore
//system/sharding/products/9
//system/sharding/products/9/e8af4de6-053b-40ad-b303-204c4ec7e395
//system/sharding/products/9/RememberEntitiesStore
//system/sharding/productsCoordinator
//system/sharding/productsCoordinator/$a
//system/sharding/productsCoordinator/singleton
//system/sharding/productsCoordinator/singleton/coordinator
//system/sharding/productsCoordinator/singleton/coordinator/RememberEntitiesStore
//system/transports
//system/transports/akkaprotocolmanager.tcp.0
//system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2FSqlSharding%40localhost%3A7918-1
//user
//user/$a
//user/$b
//user/pbm-local-clients
//user/pbm-uptime
//user/petabridge.cmd
//user/petabridge.cmd/127.0.0.1%3A54340
//user/petabridge.cmd/127.0.0.1%3A54340/handler
//user/petabridge.cmd/127.0.0.1%3A54340/handler/actor
//user/petabridge.cmd/127.0.0.1%3A54340/handler/actor/handler-0
//user/petabridge.cmd/127.0.0.1%3A54340/handler/actor/handler-0/$a
//user/petabridge.cmd-log-memorizer
//user/product-index
//user/product-index/$a
//user/product-index/product-index
//user/product-proxy
Arkatufus commented 2 years ago

The message Persistent refers to this class: https://github.com/akkadotnet/akka.net/blob/eba7bf50d4766ca7160edad6af18bdaf4970fbc7/src/core/Akka.Persistence/Persistent.cs#L297