Documentation/examples for xds resolver & balancer

[Stoakes]

Hello,

I found recently that grpc-go was working on an xds implementation for service discovery, which is great.

Using current HEAD, I tried to get both resolver and balancer working. I managed to get something more or less working for resolver by reversing code & tests code on Gist but as for balancer, I didn't.

I understand this is a quite new topic, but more documentation details or examples would help. My current blocking points are how can I configure cds or eds balancer (including how to provide xds server url and lb policy) and once Balancer is built how to use it with grpc.Dial (I didn't find much details about how to use balancer v2 API).

If I get both working and if the team sees any value in it, I would be happy to contribute a new example for xds.

Thanks

[Stoakes]

I kept on digging and eventually got a grpc service load balancing calls based on endpoints received from an Istio Pilot XDS server. 🎉

Here is the most significant snippet of it:

// dialEDS dial a service using xds api.
// targetStr format comes from Istio and must look like
// outbound|<port>||<service-name>.<namespace>.svc.cluster.local 
// Example: outbound|6565||hello-service.default.svc.cluster.local
func dialEDS(targetStr string, dialOpts ...grpc.DialOption) (*grpc.ClientConn, error) {
    return grpc.Dial(targetStr, []grpc.DialOption{
        grpc.WithBlock(),
        grpc.WithTimeout(10 * time.Second),
        grpc.WithInsecure(),
        grpc.WithDefaultServiceConfig(fmt.Sprintf(`{
            "loadBalancingConfig":[
              {
                "eds_experimental":{
                  "Cluster": "%s"
                }
              }
            ]
          }`, targetStr)),
    }...,
    )
}

Is the grpc-go team interested in an example using xds api ?

Speaking of xds, is there something planned to let user debug xdsClient state ? Something a la /debug/edsz on Istio Pilot or /config_dump on Envoy. But I guess I should open a new ticket for it.

[menghanl]

Thanks for verifying that EDS works!

The example in the original gist is how we expect xDS with gRPC to eventually work. You should only need to specify the bootstrap file, and use "xds" as the scheme. (But because xDS is still in development, "xds-experimental" is the actual scheme, and also, we have two xDS resolvers today, so "xds-experimental-new", we will delete the old one soon). The full workflow needs a new listener type as proposed in https://github.com/envoyproxy/envoy/pull/8170. I think istio may not have added support for that. That would explain why resolver seemed to be working (it actually didn't because it never got what it wants) but balancer was never started.

The new example code in your reply where Dial takes a service config with "experimental_eds" works because it bypasses LDS/RDS/CDS, and goes directly to EDS. This is an expected workflow if you are only interested in EDS (it won't have all features from the other (x)DS).

---

The xDS feature is very experimental, and I think it's a bit too early to add doc. If you want, you can write down the instructions in this issue, so other people interested can search for them. When the feature stabilizes a bit more, we will add more doc and examples.

---

For debuggability, we are adding more detailed xDS logs as the initial step. We do want to have a better way to debug. Please file an issue, and let us know if you have suggestions. Thanks!

[menghanl]

@Stoakes FYI the balancer name for EDS is changed to eds_experimental (from experimental_eds) in https://github.com/grpc/grpc-go/pull/3356. The service config in the example will need to be updated as well.

[Stoakes]

Thank you for the notice.

I was quite busy during January, so I couldn't get back on this one. I hope I can produce a working and documented example of a grpc client listening to a Pilot instance for service discovery by the middle of this week.

It will be able to switch between 2 xds balancer implementation:

• the default grpc one
• mine which pre-dates the grpc one, and contains what I'd like to have in terms of observability in grpc.

[menghanl]

A quick doc on how things are supposed to work now:

With the full LDS/RDS/CDS/EDS flow (if server supports HTTP API listener), see the client in the xds example.

---

For EDS-only mode, the client needs an xds bootstrap file (for connection to the management server) and default service config (to pick the eds balancer).

// client.go
func dial() {
    serviceConfigStr := fmt.Sprintf(`
{
  "loadBalancingConfig":[
    {"eds_experimental":{ "EDSServiceName": "%s" }}
  ]
}`, "your.service.name")

    conn, err := grpc.Dial("anything",
        grpc.WithInsecure(),
        grpc.WithDefaultServiceConfig(serviceConfigStr))
}

// bootstrap.json
{
    "xds_servers": [
        {
            "server_uri": "management-server.address"
        }
    ],
    "node": {
        "id": "12345",
        "metadata": {
            "somekey": "123456789012"
        }
    }
}

$ export GRPC_XDS_BOOTSTRAP=/path/to/bootstrap.json
$ go run client.go

[menghanl]

https://github.com/grpc/grpc-go/tree/master/examples/features/xds has the basic client/server code.

And https://istio.io/latest/blog/2021/proxyless-grpc/ would serve as a better example (with configurations) with istio