Example for fault injection

[LasseJacobs]

I was reading the proposal to add fault injection to gRPC clients: https://github.com/grpc/proposal/blob/master/A33-Fault-Injection.md

Is it possible to add an example on how to use this feature? I tried to set the environment variable which GRPC_XDS_EXPERIMENTAL_FAULT_INJECTION=true (but I don't even think that is necessary anymore since https://github.com/grpc/grpc-go/pull/4286)

But when I send a request with ones of the supported headers: newMD := metadata.Pairs("x-envoy-fault-abort-request", "400") for example, it does not work, the request just succeeds.

I tried to figure out how to use it by looking at the implementation PR: https://github.com/grpc/grpc-go/pull/4236/files but I am not able to figure it out. In particular the 'configuration' of the Listener shown in the test (https://github.com/dfawley/grpc-go/blob/2ac1aaedb828b6a73d21bb630a55d6276d2ca303/xds/internal/httpfilter/fault/fault_test.go#L520):

resources.Listeners[0].ApiListener.ApiListener = hcmAny
resources.Listeners[0].FilterChains[0].Filters[0].ConfigType = &v3listenerpb.Filter_TypedConfig{TypedConfig: hcmAny}

Should I do a similar setup for my clients? What API should I use for this?

Should this work with any Go grpc client/server? Do I need to enable support for it, set some config? Does it need to be an xDS server (https://github.com/ejona86/proposal/blob/master/A36-xds-for-servers.md)

Any input would be appreciated. I am using google.golang.org/grpc v1.55.0

[zasweq]

Hello. Fault injection in an xDS feature, so a little bit too complicated for a top level example. https://github.com/grpc/proposal/blob/master/A33-Fault-Injection.md this is the design doc cross language for the feature. It requires the sending side to use an xDS component (aka Dialing with an xds... URI). Please see design doc for specific xDS configuration required to enable the feature and the possible knobs to set through the xDS Configuration.

[LasseJacobs]

Hey, could you point me to an introductory document/example for xDS with gRPC perhaps? The document you link does not seem to do any explaining on how to actually use the feature. For example, you mention that the sending side needs to dial with an xds URI but I don't see that mentioned in the Fault injection doc.

[zasweq]

Here is an example from our codebase which tests fault injection e2e: https://github.com/grpc/grpc-go/blob/e8599844e79f461b30aec8952b27fc95325f48e8/xds/internal/httpfilter/fault/fault_test.go#L141. You can see the setup required with the management server etc., the proto resources, and the scheme required in the URI. However, this is a convenient way to test this in our repo. A real life deployment requires a control plane (e.g. https://cloud.google.com/traffic-director) that communicates with our library compiled into our binary. Here is more documentation about Proxyless Service Mesh: https://cloud.google.com/traffic-director/docs/proxyless-overview :).

[zasweq]

https://cloud.google.com/traffic-director/docs/proxyless-configure-advanced-traffic-management#fault-injection

Here is an example specifically about fault injection.

[LasseJacobs]

@zasweq perfect, thanks!