Istio creates a race condition as the app crashes with apiserver unavailable

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: yes

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

❯ istioctl version
Version: 0.7.1
GitRevision: 62110d4f0373a7613e57b8a4d559ded9cb6a1cc8
User: root@c5207293dc14
Hub: docker.io/istio
GolangVersion: go1.9
BuildStatus: Clean
❯ k version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ? Yes

What happened: Automatically injected istio sidecar broke apiserver connectivity.

What you expected to happen: Istio shouldn't introduce network faults that require manual workarounds.

How to reproduce it: This is a followup to a stash bug. Stash creates a Job to restore PV from a backup and if istio is enabled for the namespace, it breaks the stash pod. Here are the logs:

❯ k -n test-ns logs stash-recovery-test-fwmhs stash
I0507 14:54:29.776770       1 logs.go:19] FLAG: --alsologtostderr="false"
I0507 14:54:29.780598       1 logs.go:19] FLAG: --enable-analytics="false"
I0507 14:54:29.780618       1 logs.go:19] FLAG: --help="false"
I0507 14:54:29.780626       1 logs.go:19] FLAG: --kubeconfig=""
I0507 14:54:29.780636       1 logs.go:19] FLAG: --log_backtrace_at=":0"
I0507 14:54:29.780641       1 logs.go:19] FLAG: --log_dir=""
I0507 14:54:29.780647       1 logs.go:19] FLAG: --logtostderr="false"
I0507 14:54:29.780652       1 logs.go:19] FLAG: --master=""
I0507 14:54:29.780658       1 logs.go:19] FLAG: --recovery-name="test"
I0507 14:54:29.780663       1 logs.go:19] FLAG: --stderrthreshold="0"
I0507 14:54:29.780669       1 logs.go:19] FLAG: --v="3"
I0507 14:54:29.780674       1 logs.go:19] FLAG: --vmodule=""
W0507 14:54:29.780875       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0507 14:54:29.789669       1 recovery.go:41] [Get https://10.200.0.1:443/apis/stash.appscode.com/v1alpha1/namespaces/test-ns/recoveries/test: dial tcp 10.200.0.1:443: connect: connection refused]
I0507 14:54:29.792839       1 main.go:26] [Exiting Stash Main]

Notice how stash tries to talk to the apiserver (10.200.0.1:443), fails, and dies.

Here's the relevant part of istio-proxy sidecar log:

❯ k -n test-ns logs stash-recovery-test-fwmhs istio-proxy|grep 10.200.0.1_443
[2018-05-07 14:54:30.362][11][info][upstream] external/envoy/source/server/lds_api.cc:60] lds: add/update listener 'tcp_10.200.0.1_443'

As you see, envoy wasn't fast enough.

istio / old_issues_repo

Istio creates a race condition as the app crashes with apiserver unavailable #325