search

aws / aws-app-mesh-roadmap

AWS App Mesh is a service mesh that you can use with your microservices to manage service to service communication
Apache License 2.0
347 stars 25 forks source link

Implement Fault Injection #36

Open jamsajones opened 5 years ago

shubharao commented 5 years ago

Would like to hear some use case details, especially about whether there is already an fault injection mechanism that you are using

hbaldwi commented 4 years ago

I'm interested in using this feature to verify that client and service mesh deadlines/retry policies are working correctly.

bcelenza commented 4 years ago

Hey @hbaldwi, cool idea. Can you tell us a little more about how you would test and verify the deadlines and retry policies are working? Is there a specific test framework and continuous integration pipeline you're targeting to make this work?

hbaldwi commented 4 years ago

As a naive implementation, I was planning to set a small percentage of the requests to be aborted with an error, and include an integration test for client services which would send a large number of unary RPCs (I'm using GRPC) to other services, and verify that there are no errors returned to the client. The retry policy would be set such that, if it was working correctly, the probability of an error being returned to the client would be so low that it would never realistically occur.
At the very least, this would verify that our client GRPC deadlines are sufficiently high for Envoy's retry policy (given its backoff strategy), and that the retry policy is functioning correctly. I'm using Jenkins for CI, and Golang's testing package for tests.

kevinten10 commented 2 years ago

We plan to use chaos mesh for fault injection of services on appmesh