As we discovered during the course of our pains-taking bug investigation, these were, in fact, the same issue:
The ClusterSingletonManager is supposed to always belong on the oldest node of a given role type, but an original design error from the time Akka.Cluster.Tools was first introduced to Akka.NET meant that nodes were always sorted in descending order of UpNumber. This is backwards: nodes should always be sorted in ascending order of UpNumber - this means that the oldest possible node is always at the front of the "who is oldest?" list held by the ClusterSingletonManager. This explains why the singleton could appear to move early during deployments and restarts.
The ClusterSingletonManager was suspectible to a race condition where if nodes were shutdown and restarted with the same address in under 20 seconds, the default "down removal margin" used by the ClusterSingletonManager to tolerate dirty exits, it would be possible after multiple successive, fast, restarts for multiple instances of the singleton to be alive at the same time (for a short period.)
Both of these varieties of problem, duplicate singletons, is what lead to duplicate shards.
Akka.Discovery and ClusterClient Discovery Support
In Akka.NET v1.5.27 we've added support for using Akka.Cluster.Tools.ClusterClient alongside with Akka.Discovery plugins to automatically discover the initial contacts you need for ClusterClientReceptionist instances in your environment.
As we discovered during the course of our pains-taking bug investigation, these were, in fact, the same issue:
The ClusterSingletonManager is supposed to always belong on the oldest node of a given role type, but an original design error from the time Akka.Cluster.Tools was first introduced to Akka.NET meant that nodes were always sorted in descending order of UpNumber. This is backwards: nodes should always be sorted in ascending order of UpNumber - this means that the oldest possible node is always at the front of the "who is oldest?" list held by the ClusterSingletonManager. This explains why the singleton could appear to move early during deployments and restarts.
The ClusterSingletonManager was suspectible to a race condition where if nodes were shutdown and restarted with the same address in under 20 seconds, the default "down removal margin" used by the ClusterSingletonManager to tolerate dirty exits, it would be possible after multiple successive, fast, restarts for multiple instances of the singleton to be alive at the same time (for a short period.)
Both of these varieties of problem, duplicate singletons, is what lead to duplicate shards.
Akka.Discovery and ClusterClient Discovery Support
In Akka.NET v1.5.27 we've added support for using Akka.Cluster.Tools.ClusterClient alongside with Akka.Discovery plugins to automatically discover the initial contacts you need for ClusterClientReceptionist instances in your environment.
As we discovered during the course of our pains-taking bug investigation, these were, in fact, the same issue:
The ClusterSingletonManager is supposed to always belong on the oldest node of a given role type, but an original design error from the time Akka.Cluster.Tools was first introduced to Akka.NET meant that nodes were always sorted in descending order of UpNumber. This is backwards: nodes should always be sorted in ascending order of UpNumber - this means that the oldest possible node is always at the front of the "who is oldest?" list held by the ClusterSingletonManager. This explains why the singleton could appear to move early during deployments and restarts.
The ClusterSingletonManager was suspectible to a race condition where if nodes were shutdown and restarted with the same address in under 20 seconds, the default "down removal margin" used by the ClusterSingletonManager to tolerate dirty exits, it would be possible after multiple successive, fast, restarts for multiple instances of the singleton to be alive at the same time (for a short period.)
Both of these varieties of problem, duplicate singletons, is what lead to duplicate shards.
Akka.Discovery and ClusterClient Discovery Support
In Akka.NET v1.5.27 we've added support for using Akka.Cluster.Tools.ClusterClient alongside with Akka.Discovery plugins to automatically discover the initial contacts you need for ClusterClientReceptionist instances in your environment.
As we discovered during the course of our pains-taking bug investigation, these were, in fact, the same issue:
The ClusterSingletonManager is supposed to always belong on the oldest node of a given role type, but an original design error from the time Akka.Cluster.Tools was first introduced to Akka.NET meant that nodes were always sorted in descending order of UpNumber. This is backwards: nodes should always be sorted in ascending order of UpNumber - this means that the oldest possible node is always at the front of the "who is oldest?" list held by the ClusterSingletonManager. This explains why the singleton could appear to move early during deployments and restarts.
The ClusterSingletonManager was suspectible to a race condition where if nodes were shutdown and restarted with the same address in under 20 seconds, the default "down removal margin" used by the ClusterSingletonManager to tolerate dirty exits, it would be possible after multiple successive, fast, restarts for multiple instances of the singleton to be alive at the same time (for a short period.)
Both of these varieties of problem, duplicate singletons, is what lead to duplicate shards.
Akka.Discovery and ClusterClient Discovery Support
In Akka.NET v1.5.27 we've added support for using Akka.Cluster.Tools.ClusterClient alongside with Akka.Discovery plugins to automatically discover the initial contacts you need for ClusterClientReceptionist instances in your environment.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps Akka.Streams and Akka. These dependencies needed to be updated together. Updates
Akka.Streamsfrom 1.5.25 to 1.5.27.1Release notes
Sourced from Akka.Streams's releases.
... (truncated)
Changelog
Sourced from Akka.Streams's changelog.
... (truncated)
Commits
55827b7Update RELEASE_NOTES.md for 1.5.27.1 (#7306)3e2d49cFix remoting logging DefaultAddress race condition (#7305)3eacb6cAdded v1.5.28 placeholderb34d645V1.5.27 release notes (#7303)6a81fd3Akka.Cluster.Tools: deprecate ClustersSingletonManagerSettings.ConsiderAppVer...7180810Akka.Cluster.Tools: fix mutability and oldest state bugs with `ClusterSinglet...6b6afe4Add AlsoTo downstream failure propagation support (#7301)7811504Fix ActorMaterializerImpl null LogSource (#7300)f2e81c5Akka.Cluster.Tools.Singleton / Akka.Cluster.Sharding: fix duplicate shards ca...f4501e8migratedClusterSingletonManagertoswitchstatements (#7296)Updates
Akkafrom 1.5.25 to 1.5.27.1Release notes
Sourced from Akka's releases.
... (truncated)
Changelog
Sourced from Akka's changelog.
... (truncated)
Commits
55827b7Update RELEASE_NOTES.md for 1.5.27.1 (#7306)3e2d49cFix remoting logging DefaultAddress race condition (#7305)3eacb6cAdded v1.5.28 placeholderb34d645V1.5.27 release notes (#7303)6a81fd3Akka.Cluster.Tools: deprecate ClustersSingletonManagerSettings.ConsiderAppVer...7180810Akka.Cluster.Tools: fix mutability and oldest state bugs with `ClusterSinglet...6b6afe4Add AlsoTo downstream failure propagation support (#7301)7811504Fix ActorMaterializerImpl null LogSource (#7300)f2e81c5Akka.Cluster.Tools.Singleton / Akka.Cluster.Sharding: fix duplicate shards ca...f4501e8migratedClusterSingletonManagertoswitchstatements (#7296)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show