Pod HTTP Status Code not working with Istio

[fosterchuck]

BUG REPORT (Possible feature if Litmus is not intended to work with Istio Service Mesh)

What happened: When Istio Service Mesh is used between microservices, the HTTP Status Code experiment appears to run and reports completion, but does not change the actual behavior of the service under test (the status code is not updated to the test value).

What you expected to happen: Litmus runner/helper to update the IPTABLES in a manner that will allow the experiment to work with Istio sidecar. Perhaps also add a way for the experiment to report an error when it isn't actually changing traffic.

How to reproduce it (as minimally and precisely as possible):

1. Add Istio Service Mesh to at least two microservices and not the Litmus delegates, runners, helpers, etc.
2. Ensure the first microservice (service-a) is sending constant HTTP traffic to the second microservice (service-b)
3. To make the test easier have a single replica of service-a and service-b
4. Start a test targeting service-b that changes the response code (we used 401)
5. Observe either in logs of service-a, or in tracing application like APM that the response code is unchanged from what service-b provides.

Anything else we need to know?:

• By default adding Istio to the workloads results in Istio performing mTLS between the two services. Which also means that the traffic entering the second service will not arrive on the HTTP port under test, but on an Istio port (default 15006)
• In our test configuration, to avoid adding service mesh to Litmus workloads, we used annotations per workload.
• We also tested with Litmus workloads meshed, however other tests failed and the HTTP tests continued to fail, so we didn't spend much more time in this configuration.
• In our environment, this test works as expected when all other factors remain the same, but the Istio sidecars are removed from both services.
• We also tested with Litmus un-meshed, the services meshed, and the mTLS disabled between the microservices (using Istio kind: "PeerAuthentication" at the namespace level). The test also did not change the status code in this configuration.
• This article provides a very informative dissection of the IPTABLES changes that Istio performs to the NAT table: https://jimmysong.io/en/blog/sidecar-injection-iptables-and-traffic-routing/
• With the above information, we tested the following commands while the test was running (and therefore the toxiproxy should still have been running), the Status Codes were still not changed.
  • sudo iptables-legacy -t nat -I POSTROUTING -p tcp --dport 8083 -j REDIRECT --to-port 20000
  • sudo iptables-legacy -t nat -I OUTPUT -p tcp --dport 8083 -j REDIRECT --to-port 20000
  • sudo iptables-legacy -t nat -I ISTIO_OUTPUT -p tcp --dport 8083 -j REDIRECT --to-port 20000
  • sudo iptables-legacy -t nat -I ISTIO_REDIRECT -p tcp --dport 8083 -j REDIRECT --to-port 20000
• Before these changes to the IPTABLES, we validated that the REDIRECT rule that was added by the experiment did not receive packets. When adding each of the additional rules in the last bullet, they also did not receive any packets.
• We made sure to use 2.14 with the fix that mentions providing support for Service Mesh (changes "-A" to "-I" option in https://github.com/litmuschaos/litmus-go/blob/master/chaoslib/litmus/http-chaos/helper/http-helper.go) - https://github.com/litmuschaos/litmus-go/pull/578

[Feederhigh5]

Hi @fosterchuck, I am running into similar problems with my linkerd setup...

Were you able to find a workaround?

Did you only try Pod HTTP Status Code or also other network-related chaos experiments?

[fosterchuck]

We never resolved this. We switched to Linkerd Service Mesh and haven't had time to retest. If memory serves, this issue was present many months ago when we tested with Linkerd too. I cannot test the newer releases with BETA in their name, due to company restrictions. This is purely from memory, but I recall using network tests with no issue as long as they were tests that didn't rely on Toxi-Proxy. None of the HTTP tests worked. Sorry for the lack of specificity. This is not part of our current sprint, so I'm not at liberty to test. We will soon be revisiting this.

[neelanjan00]

We fixed this issue as part of this PR: https://github.com/litmuschaos/litmus-go/pull/578