Spring Reactive Load Balancer Retry Is Not Working.

[santoshkumarcts]

Hi Team,

We are implementing the spring load balancer retry feature in our project. We added RetryableLoadBalancerFilterExchangeFuntion (By Autowiring ) filter to the webclient builder by following this Reference: https://github.com/spring-cloud/spring-cloud-commons/pull/847/commits

Before this change SCLB was working fine, the traffic was distributed to all the available instances, after adding the filter, we are facing the below errors.

{"instant":{"epochSecond":1702552930,"nanoOfSecond":788156223},"thread":"boundedElastic-3","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"discovering endpoints in namespace : sampleTest","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.788+0000"} {"instant":{"epochSecond":1702552930,"nanoOfSecond":796103469},"thread":"boundedElastic-4","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"filter not present","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.796+0000"} {"instant":{"epochSecond":1702552930,"nanoOfSecond":796920350},"thread":"boundedElastic-4","level":"WARN","loggerName":"org.springframework.cloud.loadbalancer.core.RoundRobinLoadBalancer","message":"No servers available for service: 10.233.111.139","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.796+0000"} {"instant":{"epochSecond":1702552930,"nanoOfSecond":798481135},"thread":"boundedElastic-4","level":"WARN","loggerName":"org.springframework.cloud.client.loadbalancer.reactive.RetryableLoadBalancerExchangeFilterFunction","message":"LoadBalancer does not contain an instance for the service 10.233.111.139","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.798+0000"} {"instant":{"epochSecond":1702552930,"nanoOfSecond":811667123},"thread":"boundedElastic-3","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"filter not present","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.811+0000"} {"instant":{"epochSecond":1702552930,"nanoOfSecond":812302880},"thread":"boundedElastic-3","level":"WARN","loggerName":"org.springframework.cloud.loadbalancer.core.RoundRobinLoadBalancer","message":"No servers available for service: 10.233.111.139","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.812+0000"}

NOTE: Our application is using Spring Cloud Kubernetes Discovery Server, there is no implementation of ServiceInstanceListSupplier in the code. Reference : https://docs.spring.io/spring-cloud-kubernetes/reference/spring-cloud-kubernetes-discoveryserver.html#:~:text=The%20Spring%20Cloud%20Discovery%20server,the%20Service%20Account%20on%20Kubernetes.

We have the following properties added in the application.properties. spring.cloud.kubernetes.loadbalancer.mode= POD spring.cloud.loadbalancer.ribbon.enabled= false spring.cloud.loadbalancer.retry.enabled= true spring.cloud.loadbalancer.retry.retry-on-all-exceptions= true spring.cloud.loadbalancer.retry.max-retries-on-next-service-instance=2 spring.cloud.loadbalancer.retry.max-retries-on-same-service-instance=0 spring.cloud.loadbalancer.retry.retry-on-all-operations= true

Please let me know if any additional information is needed.

Thank you

[OlgaMaciaszek]

Hello @santoshkumarcts. Thanks for reporting the issue. Please provide a minimal, complete, verifiable example that reproduces the issue. Please make sure it's against Spring Cloud 2023.0.0, as currently that's the only version with OSS support.

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.