Cannot find service - SERVICE_UNAVAILABLE

[rcbandit111]

I created a test project with Spring Cloud gateway and Spring microservice which are using implementation 'org.springframework.cloud:spring-cloud-kubernetes-client-discovery':

https://github.com/rcbandit111/kubernetes_discovery_client_poc

I configured rotes into Spring Gateway file:

https://github.com/rcbandit111/kubernetes_discovery_client_poc/blob/main/gateway/src/main/resources/application.yml

Both services are deployed into Kubernetes but request is not forwarded:

2024-04-02 13:21:55.817 TRACE 1 --- [     parallel-1] o.s.c.g.filter.RouteToRequestUrlFilter   : RouteToRequestUrlFilter start
2024-04-02 13:21:55.886 TRACE 1 --- [     parallel-1] o.s.c.g.filter.GatewayMetricsFilter      : spring.cloud.gateway.requests tags: [tag(httpMethod=GET),tag(httpStatusCode=503),tag(outcome=SERVER_ERROR),tag(routeId=microservice_1),tag(routeUri=lb://microservice_1),tag(status=SERVICE_UNAVAILABLE)]
2024-04-02 13:22:42.084 TRACE 1 --- [     parallel-1] o.s.c.g.f.WeightCalculatorWebFilter      : Weights attr: {}
2024-04-02 13:22:42.089 TRACE 1 --- [     parallel-1] o.s.c.g.h.p.PathRoutePredicateFactory    : Pattern "/api/microservice_1/dashboard/users" matches against value "/api/microservice_1/dashboard/users"
2024-04-02 13:22:42.095 DEBUG 1 --- [     parallel-1] o.s.c.g.h.RoutePredicateHandlerMapping   : Route matched: microservice_1
2024-04-02 13:22:42.096 DEBUG 1 --- [     parallel-1] o.s.c.g.h.RoutePredicateHandlerMapping   : Mapping [Exchange: GET http://11.111.63.1:30057/api/microservice_1/dashboard/users?startDate=2024-03-02T14:21:45.540Z&endDate=2024-04-02T13:22:40.533Z] to Route{id='microservice_1', uri=lb://microservice_1, order=0, predicate=Paths: [/api/microservice_1/dashboard/users], match trailing slash: true, gatewayFilters=[[[RemoveRequestHeader name = 'Cookie'], order = 1], [[SpringCloudCircuitBreakerResilience4JFilterFactory name = 'microservice_1', fallback = [null]], order = 2], [[RewritePath /api/dashboard/users = '/dashboard/users'], order = 3]], metadata={}}
2024-04-02 13:22:42.096 DEBUG 1 --- [     parallel-1] o.s.c.g.h.RoutePredicateHandlerMapping   : [9b4aa35e-33] Mapped to org.springframework.cloud.gateway.handler.FilteringWebHandler@20a20ac5
2024-04-02 13:22:42.097 DEBUG 1 --- [     parallel-1] o.s.c.g.handler.FilteringWebHandler      : Sorted gatewayFilterFactories: [[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter@39c1fe0b}, order = -2147483648], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@b016b4e}, order = -2147482648], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@3e5d4f6b}, order = -1], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter@367b22e5}, order = 0], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter@10bd9df0}, order = 0], [[RemoveRequestHeader name = 'Cookie'], order = 1], [[SpringCloudCircuitBreakerResilience4JFilterFactory name = 'microservice_1', fallback = [null]], order = 2], [[RewritePath /api/dashboard/users = '/dashboard/users'], order = 3], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@5ae95707}, order = 10000], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.config.GatewayNoLoadBalancerClientAutoConfiguration$NoLoadBalancerClientFilter@3ae0b770}, order = 10150], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@fca387}, order = 2147483646], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@1ac45389}, order = 2147483647], [GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter@62891fc8}, order = 2147483647]]

I suspect that microservice name is not resolved. Any idea what might be wrong?

[rcbandit111]

@wind57 Do I need to make a additional configuration in order resolve remote service pod?

[wind57]

I'm sorry, but I can't help you too much, because 1) that is an out dated version of spring-cloud-kubernetes-client-discovery that we do not support anymore 2) I have not worked too much with gateway to be able to debug this properly.

[wind57]

Can you try something different to begin with? You can auto-wire the DiscoveryClient and call its methods on it to see what is discoverable to begin with. Does that make sense?

[rcbandit111]

@wind57 I tried with the latest version 'org.springframework.cloud:spring-cloud-kubernetes-client-discovery:3.1.1' But still I get SERVICE_UNAVAILABLE

You can auto-wire the DiscoveryClient and call its methods on it to see what is discoverable to begin with. Does that make sense?

Ok but can you show me how with some very basic code snippet?

For example how I can get as a list populated hostname and internal address of a pod?

[wind57]

it is a simple as @Autowire (on better constructor inject) a DiscoveryClient and then calling its method on it: getServices (this will give you the serviceIds) and with each of them call getInstances to get the detailed information on each of them.

[rcbandit111]

@wind57 I tried this:

    DiscoveryClient discoveryClient;

    @Autowired
    public ServiceRouteConfiguration(DiscoveryClient discoveryClient) {
        this.discoveryClient = discoveryClient;
    }

    @EventListener(ApplicationReadyEvent.class)
    public void test(){
        int size = discoveryClient.getServices().size();
        System.out.println("Kubernetes services size " + size);
        discoveryClient.probe();
        for (String service : discoveryClient.getServices()) {
            System.out.println("Discovered service " + service);
        }

    }

And the result is only:

Kubernetes services size 0

Looks like data cannot be extracted?

[wind57]

but where are your services? what does kubectl get services --all-namespaces tell you?

[rcbandit111]

@wind57 I don't see a list of services. Only the output that I show you. I use native Kubernetes. v1.29.2. I get this from the command kubectl get services --all-namespaces:

root@node1:~# kubectl get services --all-namespaces
NAMESPACE     NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
default       microservice_1                            NodePort       10.233.21.4     <none>        8761:30118/TCP                                 4m
default       gateway                                   LoadBalancer   10.233.59.225   <pending>     8888:30057/TCP                                 10m
default       kubernetes                                ClusterIP      10.233.0.1      <none>        443/TCP                                        26d
default       mockup                                    LoadBalancer   10.233.3.176    <pending>     8761:30066/TCP                                 37h
kafka         dev-cluster-kafka-bootstrap               ClusterIP      10.233.20.252   <none>        9091/TCP,9092/TCP,9093/TCP                     26d
kafka         dev-cluster-kafka-brokers                 ClusterIP      None            <none>        9090/TCP,9091/TCP,8443/TCP,9092/TCP,9093/TCP   26d
kafka         dev-cluster-zookeeper-client              ClusterIP      10.233.11.208   <none>        2181/TCP                                       26d
kafka         dev-cluster-zookeeper-nodes               ClusterIP      None            <none>        2181/TCP,2888/TCP,3888/TCP                     26d
kube-system   coredns                                   ClusterIP      10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP                         26d
newrelic      newrelic-bundle-kube-state-metrics        ClusterIP      10.233.52.246   <none>        8080/TCP                                       25d
newrelic      newrelic-bundle-nri-metadata-injection    ClusterIP      10.233.53.22    <none>        443/TCP                                        25d
root@node1:~#

[wind57]

something is off here... the example you provided is not reproducible for me to be able to look at it. You do not have even manifests there in order for me to deploy the service.

If you can provide an example, I would take a closer look. Here is an example what I mean : https://stackoverflow.com/help/minimal-reproducible-example

[rcbandit111]

ok, I will provide one with Helm chart deployment later today. Do you have available Kubernetes cluster?

[rcbandit111]

@wind57 Here is the example https://github.com/rcbandit111/mockup You can find the helm chart into the sub folder?

[wind57]

I don't need those helm manifests (I can't go through them and understand all the templating that you have in place), just create plain deployment/service files and provide clear instructions on how to deploy those, in kind for example. Then, I can replicate them in my local kind cluster

[rcbandit111]

@wind57 I pushed the mockup project into a public docker.io registry.

I can run kubectl run mockup-pod --image=peterpenzov/mockup:1c9c95f9039e. Pod is created. Please try it.

P.S. I tried with Spring Cloud 2023.0.1. It's working fine. Any chance to make it work with Spring Cloud 2021.0.9?

[wind57]

No, that is a version we do not support anymore. You can close this issue if its working for you.

[rcbandit111]

Can you propose some other alternative solution?

[wind57]

not really, as said numerous times, we don't support that version anymore for a lot of time, so nothing can be done, imho.

[wind57]

I kind of felt bad that we do not have a sample, where you can run with gateway and our project when you opened this issue, so I decided to create one. See this: https://github.com/wind57/gateway-demo