XDS listener moving to NOT_SERVING

[costinm]

What version of gRPC are you using?

master

What version of Go are you using (go version)?

1.15

What operating system (Linux, Windows, …) and version?

Linux, Debian (gDebian)

What did you do?

Proxyless gRPC, using Istio echo client/server

What did you expect to see?

First request works - server starts, client connects and returns expected result.

I am then connecting to an invalid/not working destination - it fails as expected. After that - the good destination fails.

It appears the listener gets stopped because somehow the LDS response lacks the inbound listener - however on Istiod we log the number of listeners sent, and it is correct.

I'm still debugging

Enabled verbose logging:

(normal XDS config for listener)

{"level":"error","ts":1652467715.5276887,"caller":"xds/server.go:156","msg":"[xds][xds-server 0xc0000a45a0] Listener \"[::]:17070\" entering mode: \"SERVING\"","system":"grpc","grpc_log":true,"stacktrace":"google.golang.org/grpc/xds.(*GRPCServer).loggingServerModeChangeCallback\n\t/ws/istio.io/grpc/xds/server.go:156\ngoogle.golang.org/grpc/xds.(*GRPCServer).handleServingModeChanges\n\t/ws/istio.io/grpc/xds/server.go:321\ngoogle.golang.org/grpc/xds.(*GRPCServer).Serve.func1\n\t/ws/istio.io/grpc/xds/server.go:247"}

( normal CDS/EDS for the good service )

( CDS for the bad service ) 
{"level":"info","ts":1652467826.8737702,"caller":"xdsresource/unmarshal_lds.go:63","msg":"Resource with name: echo-grpc-v1.echo-grpc.svc.cluster.local:7070, type: *listenerv3.Listener, contains: {\n  \"na
me\": \"echo-grpc-v1.echo-grpc.svc.cluster.local:7070\",\n  \"address\": {\n    \"socketAddress\": {\n      \"address\": \"10.0.4.239\",\n      \"portValue\": 7070\n    }\n  },\n  \"apiListener\": {\n    
\"apiListener\": {\n      \"@type\": \"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\n      \"rds\": {\n        \"configSource\": {\n          \"
ads\": {\n\n          }\n        },\n        \"routeConfigName\": \"outbound|7070||echo-grpc-v1.echo-grpc.svc.cluster.local\"\n      },\n      \"httpFilters\": [\n        {\n          \"name\": \"envoy.fi
lters.http.fault\",\n          \"typedConfig\": {\n            \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault\"\n          }\n        },\n        {\n          \"name\": 
\"envoy.filters.http.router\",\n          \"typedConfig\": {\n            \"@type\": \"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\"\n          }\n        }\n      ]\n    }\n  }\n}"
,"system":"grpc","grpc_log":true}

{"level":"error","ts":1652467826.8744009,"caller":"xds/server.go:158","msg":"[xds][xds-server 0xc0000a45a0] Listener \"[::]:17070\" entering mode: \"NOT_SERVING\" due to error: xds: ListenerResource target xds.istio.io/grpc/lds/inbound/[::]:17070 not found in received response, xDS client nodeID: id:\"sidecar~10.48.1.174~echo-grpc-v1-94c9f5bd-wlp9z.echo-grpc~echo-grpc.svc.cluster.local\"   "stacktrace":"google.golang.org/grpc/xds.(*GRPCServer).loggingServerModeChangeCallback\n\t/ws/istio.io/grpc/xds/server.go:158\ngoogle.golang.org/grpc/xds.(*GRPCServer).handleServingModeChanges\n\t/ws/istio.io/grpc/xds/server.go:321\ngoogle.golang.org/grpc/xds.(*GRPCServer).Serve.func1\n\t/ws/istio.io/grpc/xds/server.go:247"}

...

[costinm]

It looks like the problem is in Istiod ( or in both ) - when gRPC adds a new cluster to the list, Istiod will detect nothing change and only send the delta. Not yet clear what is the correct behavior - long term delta seems far more efficient - but will fix istiod.