Aggregate Command handlers not Discoverable with Distributed Command Bus after 4.3.2 upgrade

[shubhojitr]

After upgrading axon-spring-boot-starter version from 4.3 to 4.3.2, all the Aggregate command handlers are not discoverable and throwing "No node known to accept" error. However Non Aggregate command handlers are able to handle in both 4.3 and 4.3.2 version.

Below is the axon stack I'm using to replicate this issue.

axon-spring-boot-starter: 4.3.2 axon-springcloud-spring-boot-starter: 4.3

Please note that this is not hapenning once I downgrade axon-spring-boot-starter from 4.3.2 to 4.3

org.axonframework.commandhandling.NoHandlerForCommandException: No node known to accept [com.roy.axon.cmd.IssueCardCommand] at org.axonframework.commandhandling.distributed.DistributedCommandBus.dispatch(DistributedCommandBus.java:184) ~[axon-messaging-4.3.2.jar:4.3.2] at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:75) ~[axon-messaging-4.3.2.jar:4.3.2] at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:73) ~[axon-messaging-4.3.2.jar:4.3.2] at org.axonframework.commandhandling.gateway.DefaultCommandGateway.sendAndWait(DefaultCommandGateway.java:90) ~[axon-messaging-4.3.2.jar:4.3.2] at com.roy.axon.controller.GiftCardController.issueGiftCard(GiftCardController.java:24) ~[classes/:na]

[smcvb]

That's definitely not desirable behaviour. Can you tell me whether the same faulty behaviour occurs when you use Axon Framework version 4.3.1? That'll narrow down the possible "issue introduction commits" quite some.

Added, it would also be highly beneficial if you have a sample project available showing this issue. I am guessing you have something ready as the stack trace shows GiftCardController, which signals you might be using Axon's GiftCard demo project to showcase the problem.

[smcvb]

I've taken the effort to produce a sample application testing this issue you've filed @shubhojitr. Just wanted to let you know I've reached the point that I am noticing the same behaviour. Looking at what the problem might be as we speak.

By the way, what would be valuable for me to know is which Spring Cloud implementation you are using in your set up. Is it Eureka? Consul? Kubernetes?

[smcvb]

Alright, it proofs to be rather important to know the version of Spring Cloud you are using @shubhojitr.

I've been able to reproduce this twice whilst using Eureka as the discovery mechanism, but only by sheer chance. The SpringCloudCommandRouter defaults to updating a ServiceInstance's metadata, which in Eureka's case goes through only prior to the local ServiceInstance being fully registered.

Solution on this end would be to enforce the the fallback solution of the SpringCloudCommandRouter, as is mentioned in this issue. As a quick local fix to ensure the mentioned issue isn't the culprit, I've build a snapshot version of the extension enforcing the fallback approach.

However, this issue turns up regardless of whether I am using Axon Framework 4.3, 4.3.1 or 4.3.2. I am thus uncertain how to further replicate the problem you are encountering.

The status of this issue will thus be adjusted back to "Information Required".

[shubhojitr]

@smcvb , just did a quick test and and you are right it's happening in 4.3, 4.3.1 and 4.3.2. Sometimes the very first command is handled and thereafter it's not able to find any command handler.

I faced this issue while upgrading axon version from 4.2 to 4.3.2 in an existing axon application.

today I tried to replicate this issue with spring-boot 2.1.11.RELEASE and spring-cloud Greenwich.SR2 and spring-boot 2.2.5.RELEASE and spring-cloud Hoxton.SR3. I was able to reproduce org.axonframework.commandhandling.NoHandlerForCommandException: No node known to accept on both occasion(keeping axon-spring-boot and axon-springcloud 4.3 as constant)

I could not dig deep due to other commitments but will definitely try to debug more in case I get some time.

Thanks, Roy

[shubhojitr]

I've taken the effort to produce a sample application testing this issue you've filed @shubhojitr. Just wanted to let you know I've reached the point that I am noticing the same behaviour. Looking at what the problem might be as we speak.

By the way, what would be valuable for me to know is which Spring Cloud implementation you are using in your set up. Is it Eureka? Consul? Kubernetes?

I'm using Eureka as discovery mechanism.

[smcvb]

Thanks for the info here @shubhojitr, that definitely clarifies things.

When it comes to versioning, the Axon Spring Cloud Extension is currently not using any of the Spring Cloud release train dependencies. It directly depends on the spring-cloud-commons module, currently using version 2.0.1.RELEASE. The Spring Boot version being used is 2.1.2.RELEASE.

Tracking down this page points out that Greenwich.SR2 contains spring-cloud-commons 2.1.2.RELEASE. For Hoxton I found this page, which notes that Hoxton.SR3 contains spring-cloud-commons 2.2.2.RELEASE. The aforementioned sample app I created runs with 2.1.2.RELEASE (thus Greenwich.SR2), but it didn't with 2.2.2.RELEASE.

The most important hint you are giving me as that you are using Eureka as the discovery mechanism. Tracking back to my previous comment, I said the following:

I've been able to reproduce this twice whilst using Eureka as the discovery mechanism, but only by sheer chance. The SpringCloudCommandRouter defaults to updating a ServiceInstance's metadata, which in Eureka's case goes through only prior to the local ServiceInstance being fully registered.

Solution on this end would be to enforce the the fallback solution of the SpringCloudCommandRouter, as is mentioned in this issue. As a quick local fix to ensure the mentioned issue isn't the culprit, I've build a snapshot version of the extension enforcing the fallback approach.

The fact you are using Eureka at the moment is the issue why it's not working... Eureka's discovery service is successfully able to update a ServiceInstance's metadata once. The first update will always correspond to the CommandFilter DenyAll, meaning that the information shared will stay put it telling all the nodes that it cannot handle any command.

As mentioned too, this can be fixed by not having a back-up solution like the SpringCloudHttpBackupCommandRouter, but by forcefully using the mechanism employed by the SpringCloudHttpBackupCommandRouter. The key here is the SpringCloudHttpBackupCommandRouter#getMessageRoutingInformation(ServiceInstance) method on line 133.

As a quick fix on your end, you can change that method to the following:

@Override
protected Optional<MessageRoutingInformation> getMessageRoutingInformation(ServiceInstance serviceInstance) {
    return requestMessageRoutingInformation(serviceInstance);
}

This adjustment means that the SpringCloudHttpBackupCommandRouter will no longer first check for existing metadata to know which commands a service can handle. Instead, it will request the CommandFilter information by doing a dedicated rest call. Note that for this, you will need to enable the back up solution. This can be done through by setting the axon.distributed.spring-cloud.fallback-to-http-get application property to true.

Do know that the easiest solution for you would be to wait for issue #17 to be resolved for the Axon Spring Cloud extension. And lastly, this also means that this isn't an Axon Framework issue, but an extension problem. As such, I will be closing this issue as obsolete. I'd recommend to track #17 for progress on that part Roy.

[shubhojitr]

Thank you @smcvb for the detailed explanation. For now I'm going to stick to 4.2 version and wait for the fix in axon spring-cloud extension. However I'm definitely going to try the workaround you provided.