CommandRouter Re-registers with Eureka even if the status = DOWN

[manishatGit]

Basic information

• Axon Framework version: axon-bom:4.9.4'
• JDK version: 21
• Spring Cloud Extension version:
• Complete executable reproducer if available (e.g. GitHub Repo): None
• Running in Application in K8s environment with 4 Application pods connecting to Eureka discovery service running in a separate pod.

Steps to reproduce

1. Run Axon Application with more than three nodes
2. Make sure all the nodes are Registered to Eureka successfully
3. Ingest at least 50K commands to the Axon command gateway
4. Kill all the instances at the same time.
5. Watch the number of Registered instances with Eureka at: http://eureka-host:8761
6. There will be at least one instance still registered with Eureka in emergency state (self-preservation-mode)
7. Other nodes will be routing commands to this node

Expected behavior

As soon as the Axon application dies it should deregister itself with Eureka Service and never re-register until the next heartbeat event.

Actual behavior

When an application shuts down, it de-registers itself with the Eureka server with the status DOWN and registers itself again without having the status UP with the application. This is causing the dead application instance to still be in the list of members to which commands can be routed for some time and eventually removed. The application should immediately de-register and never register in case of an application shutdown event.

[abuijze]

I've investigated this issue, and came to the conclusion that this issue can't be Axon related. Axon only reads from the discovery service (upon each heartbeat) to update its internal view of what the capabilities of the nodes around it are. It fetches those capabilities by calling an endpoint on the services that are published as being available.

Self-preservation mode is a feature of Eureka to prevent services from being marked as unavailable in case of a network failure or glitch.

Spring Cloud Discovery does te registration or deregistration of services with the Discovery Server. It might be that there is something at fault there.

If you have any indication that Axon does influence the re-registration of nodes, feel free to commend and reopen the ticket.