search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.46k stars 3.85k forks source link

Cannot get the example-xds to work #9639

Closed alesj closed 1 year ago

alesj commented 2 years ago

I'm trying to get examples/example-xds to work.

I'm using Istio as XDS management server:

This is my PoC app (copied the example's code)

    public void startServer(@Observes StartupEvent evt) throws Exception {
        try (InputStream stream = getClass().getClassLoader().getResourceAsStream("bootstrap.json")) {
            byte[] xdsConfig = stream.readAllBytes();
            System.setProperty("io.grpc.xds.bootstrapConfig", new String(xdsConfig));
        }
        ServerCredentials credentials = InsecureServerCredentials.create();
        // Since the main server may be using TLS, we start a second server just for plaintext health
        // checks
        new Thread(() -> {
            try {
                server = XdsServerBuilder.forPort(port, credentials)
                    .addService(new HelloService())
                    .addService(ProtoReflectionService.newInstance()) // convenient for command line tools
                    .build()
                    .start();
                log.info("Listening on port " + port);
                server.awaitTermination();
            } catch (Exception e) {
                log.error("Error: ", e);
                Quarkus.asyncExit();
            }
        }, "xDS Server").start();
    }

I then build a Docker image out of this PoC app

And deploy it into the same namespace as Istio

My app then gets stuck on this log msg

The 'gRPC server' started (in the log) is another Quarkus gRPC server instance, which shouldn't be an issue. (I'm using Quarkus's gRPC support to simplify some of the stuff)

Afaics my xDS server can reach Istio, but not the other way around, right? Or what is that WARNING trying to tell me?

My node in bootstrap.json looks like this:

  "node": {
    "id": "router~0.0.0.0~myid~xyz",
    "cluster": "foobar"
  },

as I had to work around the error of not having 4 part, etc ...

Could this be an issue?

alesj commented 2 years ago

Modifying things a bit ... trying to push in the right IP, instead of 0.0.0.0 or localhost ... but still no success. e.g. 

        try (InputStream stream = getClass().getClassLoader().getResourceAsStream("bootstrap.json")) {
            byte[] xdsBytes = stream.readAllBytes();
            String xdsConfig = new String(xdsBytes);
            String ip = InetAddress.getLocalHost().getHostAddress();
            log.info("Current IP: {}", ip);
            xdsConfig = xdsConfig.replace("${ip}", ip);
            System.setProperty("io.grpc.xds.bootstrapConfig", xdsConfig);
        }
YifeiZhuang commented 2 years ago

How is the xds management server configured on LDS resource information? It looks xDS server is reaching Istio, but no resource is returned. The UNAVAILABLE error code is due to (~15s) timeout when xds client does not receive any response and retried the stream towards istio. So it notified onAbsent.

alesj commented 2 years ago

@YifeiZhuang all I do is run that ./bin/istioctl install --set profile=minimal which should setup Istio in K8s (Minikube in my case). Afaiwt that should also setup xDS ... Dunno if that's enough? @luksa ^ any thoughts? :-)

sanjaypujare commented 2 years ago

@alesj we recently added support for Istio with grpc-java. Istio + proxyless-gRPC is described in this blog https://istio.io/latest/blog/2021/proxyless-grpc/ . Although the blog covers an example with grpc-go the same concepts apply to Java. Things to note:

There is dependency on the XdsClient supporting communication over UDS with Istio's xDS proxy. This was recently added to Java but is yet to be tested with Istio.

For server side especially make sure you follow the instructions in the blog so the appropriate Listener is returned to the server. Please let us know what you find.

alesj commented 2 years ago

@sanjaypujare yeah, saw that blog before, and the examples/example-cds. But what I was missing is an example of bootstrap.json that should be used.

Do I really need to follow all those k8s steps? Or why isn't ./bin/istioctl install --set profile=minimal + my app k8s deployment enough? (or what I might be missing on the Istio side / deployment?)

Do I also need to change how my proto service is built?

alesj commented 2 years ago

But what I was missing is an example of bootstrap.json that should be used.

e.g. I had to check Istio code to see how the node's id should look like -- 4 parts split with ~

alesj commented 2 years ago

Istio log after the "server" app tries to connect:

alesj commented 2 years ago

OK, typo in port name ... now:

Skywalker:quarkus alesj$ kubectl get endpoints qcl-app -n istio-system NAME ENDPOINTS AGE qcl-app 172.17.0.4:30051 5d23h

But still I get this ...

2022-10-26T07:59:46.599064Z info ads Incremental push, service qcl-app.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints 2022-10-26T07:59:46.747565Z info ads Push debounce stable[7] 3 for config ServiceEntry/istio-system/qcl-app.istio-system.svc.cluster.local: 135.307083ms since last change, 148.409042ms since last push, full=false 2022-10-26T07:59:46.747715Z info ads XDS: Incremental Pushing:2022-10-26T07:07:49Z/2 ConnectedEndpoints:0 Version:2022-10-26T07:07:49Z/2 2022-10-26T07:59:48.756881Z info ads Push debounce stable[8] 3 for config ServiceEntry/istio-system/qcl-app.istio-system.svc.cluster.local: 101.014209ms since last change, 138.915417ms since last push, full=false 2022-10-26T07:59:48.756985Z info ads XDS: Incremental Pushing:2022-10-26T07:07:49Z/2 ConnectedEndpoints:0 Version:2022-10-26T07:07:49Z/2

alesj commented 2 years ago

Finally progress ... Was missing "injection":

Now it all starts as it should:

2022-10-28 11:11:54,246 FINER [io.grp.xds.XdsLogger] (grpc-default-executor-0) [xds-client<10>: (unix:///etc/istio/proxy/XDS)] Sending ACK for LDS update, nonce: zd+Nre/rvXw=30965a49-3ec0-4b4e-9ee1-ccedf272bd7c, current version: 2022-10-27T12:47:09Z/4 2022-10-28 11:11:54,364 FINER [io.grp.xds.XdsServerWrapper] (grpc-default-executor-0) Delegate server started. 2022-10-28 11:11:54,386 INFO [com.ale.qcl.app.Configuration] (xDS Server) Listening on port 30051

And the bootstrap.json is referenced / magically added via:

Environment: GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true GRPC_XDS_BOOTSTRAP: /etc/istio/proxy/grpc-bootstrap.json

sanjaypujare commented 2 years ago

Now it all starts as it should:

Great! Does it work end to end as expected? Do you have both client and server configured as proxyless so using the Istio bootstrap file? Would it be possible to see your code and config somewhere? Thx

alesj commented 2 years ago

Finally got it working on both ends -- the client side then was easy, just had to code it and properly direct it to the target / server.

The code is here (as mentioned in the initial description)

It has some Quarkus magic: DI, Docker image build (via JIB), etc

Main code - single class :-)

My k8s resources:

alesj commented 2 years ago

And this is how it looks when Quarkus does all the magic behind the scene :-)

Instead of plain gRPC server Quarkus creates XDS based wrapper / server. And all the stubs get XDS channel.

# XDS
quarkus.grpc.server.xds.enabled=true
quarkus.grpc.server.port=30051

quarkus.grpc.clients.stub.name-resolver=xds
quarkus.grpc.clients.stub.host=qcl-app
quarkus.grpc.clients.stub.port=30051
sanjaypujare commented 1 year ago

If there is no issue, can we close this issue?