There were multiple issues, but basically the fact that we would not store the routed subscriptions with the origin of the LEAF they came from made the server unable to differentiate those compared to "local" routed subscriptions, which in some cases (like a server restart and the resend of subscriptions) could lead to servers sending incorrectly subscription interest to leaf connections.
We are now storing the subscriptions with a sub type indicator and the origin (for leaf subscriptions) as part of the key. This allows to differentiate "regular" routed subs versus the ones on behalf of a leafnode.
An INFO boolean is added LNOCU to indicate support for origin in the LS- protocol, which is required to properly handle the removal. Therefore, if a route does not have LNOCU, the server will behave like an old server, and store with the key that does not contain the origin, so that it can be removed when getting an LS- without the origin. Note that in the case of a mix of servers in the same cluster, some of the issues this PR is trying to fix will be present (since the server will basically behave like a server without the fix).
Having a different routed subs for leaf connections allow to revisit the fix #5982 that was done for issue #5972, which was about a more fair queue distribution to a cluster of leaf connections. That fix actually introduced a change in that we always wanted to favor queue subscriptions of the cluster where the message is produced, which that fix possibly changed. With this current PR, the server can now know if a remote queue sub is for a "local" queue sub there or on behalf of a leaf and therefore will not favor that route compared to a leaf subscription that it may have directly attached.
There were multiple issues, but basically the fact that we would not store the routed subscriptions with the origin of the LEAF they came from made the server unable to differentiate those compared to "local" routed subscriptions, which in some cases (like a server restart and the resend of subscriptions) could lead to servers sending incorrectly subscription interest to leaf connections.
We are now storing the subscriptions with a sub type indicator and the origin (for leaf subscriptions) as part of the key. This allows to differentiate "regular" routed subs versus the ones on behalf of a leafnode. An INFO boolean is added
LNOCUto indicate support for origin in theLS-protocol, which is required to properly handle the removal. Therefore, if a route does not haveLNOCU, the server will behave like an old server, and store with the key that does not contain the origin, so that it can be removed when getting an LS- without the origin. Note that in the case of a mix of servers in the same cluster, some of the issues this PR is trying to fix will be present (since the server will basically behave like a server without the fix).Having a different routed subs for leaf connections allow to revisit the fix #5982 that was done for issue #5972, which was about a more fair queue distribution to a cluster of leaf connections. That fix actually introduced a change in that we always wanted to favor queue subscriptions of the cluster where the message is produced, which that fix possibly changed. With this current PR, the server can now know if a remote queue sub is for a "local" queue sub there or on behalf of a leaf and therefore will not favor that route compared to a leaf subscription that it may have directly attached.
Resolves #5972 Resolves #6148
Signed-off-by: Ivan Kozlovic ivan@synadia.com