Istio Bookinfo: upstream connect error or disconnect/reset before headers

[redlifejacket]

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: YES

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version: Not installed
kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-gke.1", GitCommit:"4ce7af72d8d343ea2f7680348852db641ff573af", GitTreeState:"clean", BuildDate:"2018-01-31T22:30:55Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ? NO Did you install istio.yaml, istio-auth.yaml.... NO What happened: As per the Quick Start with Google Kubernetes Engine, I have installed the BookInfo application using the Istio GKE Deployment Manager.

The istio-cluster is up and the workloads are running. gcloud container clusters list NAME ZONE MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS istio-cluster asia-southeast1-a 1.9.2-gke.1 35.186.152.80 n1-standard-2 1.9.2-gke.1 4 RUNNING

kubectl get deployments,ing NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deploy/details-v1 1 1 1 1 1h deploy/productpage-v1 1 1 1 1 1h deploy/ratings-v1 1 1 1 1 1h deploy/reviews-v1 1 1 1 1 1h deploy/reviews-v2 1 1 1 1 1h deploy/reviews-v3 1 1 1 1 1h

NAME HOSTS ADDRESS PORTS AGE ing/gateway * 35.197.150.179 80 1h

kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS AGE gateway * 35.197.150.179 80 1h

When I access http://35.197.150.179/productpage I get: upstream connect error or disconnect/reset before headers I can access the Istio dashboard at http://localhost:3000/dashboard/db/istio-dashboard

I can access the Prometheus Graph at: http://localhost:9090/graph

I can access the ServiceGraph at: http://localhost:8088/dotviz

I can access the Tracing at: http://localhost:9411/zipkin/?serviceName=all&spanName=all&lookback=3600000&startTs=1518598745092&endTs=1518602345092&annotationQuery=&minDuration=&limit=10&sortOrder=duration-desc

Could you please shed light. What you expected to happen:

How to reproduce it: When I access http://35.197.150.179/productpage I get: upstream connect error or disconnect/reset before headers

Feature Request: Y/N

Describe the feature:

[redlifejacket]

/cc @salrashid123 @selmanj @ldemailly

[salrashid123]

I'm an owner now of the target GCP project and saw the sidecars didn't get installed though the other components did

$ kubectl get namespace -L istio-injection
NAME           STATUS    AGE       ISTIO-INJECTION
default        Active    10h       enabled
istio-system   Active    10h       
kube-public    Active    10h       
kube-system    Active    10h       

$ kubectl get no,ing,po,ing,deployments -n istio-system
NAME                                         READY     STATUS    RESTARTS   AGE
po/grafana-648859cf87-nlsbq                  1/1       Running   0          10h
po/istio-ca-797dfb66c5-xh9kz                 1/1       Running   0          10h
po/istio-ingress-84f75844c4-nfftm            1/1       Running   0          10h
po/istio-mixer-9bf85fc68-r7m5w               3/3       Running   0          10h
po/istio-pilot-575679c565-wwnlb              2/2       Running   0          10h
po/istio-sidecar-injector-7b559f7f6f-kmpxp   1/1       Running   0          10h
po/prometheus-cf8456855-lwp4p                1/1       Running   0          10h
po/servicegraph-7ff6c499cc-2r94n             1/1       Running   0          10h
po/zipkin-7988c559b7-dqjqn                   1/1       Running   0          10h

kubectl get po
NAME                                 READY     STATUS    RESTARTS   AGE
po/details-v1-64b86cd49-w4jfc        1/1       Running   0          10h
po/productpage-v1-84f77f8747-fm8km   1/1       Running   0          10h
po/ratings-v1-5f46655b57-9nx5t       1/1       Running   0          10h

I deleted bookinfo an dthen redeployed the app and saw the sidecars pick up.

suspect this maybe a timing issue on our GCP DM script (i.,e we run through istio init, webhook setup for 0.5.1 then wait ~10s before deploying bookinfo.

I can look into some watch command or something that may indicate the sidecar injection is ready? Any thoughts on what to look for?

[ldemailly]

@selmanj can you have a look?

[johnzheng1975]

This issue still exists in istio0.5.1(0.7.1) + k8s 1.9.2 https://github.com/istio/istio/issues/4999

[ldemailly]

is mtls on ?

[johnzheng1975]

@ldemailly No, we only applied istio.yaml, not applied istio-auth.yaml

[farcaller]

Same problem ("upstream connect error or disconnect/reset before headers") but with auth enabled.

[aneslozo]

+1 Same issue, auth enabled, istio 0.7.1 and k8s 1.8.9.

Tested locally with k8s 1.9.4 and istio 0.7.1 using istio-auth.yaml and didn't have this issue.

[ejether]

Also same problem using k8s 1.10.2-gke.1 , istio 0.7.1 auth enabled and istio-injector configured.

[hackintoshrao]

Facing this issue on GKE 1.9 and istio 0.u71 auth enabled.

[costinm]

We probably need some feedback on the status of Istio - so installers (and automated tests) can wait until istio is fully deployed. Having the istio pods running is not sufficient - auto-injector depends on certificates, which depend on volume propagation. Istio is ready when all components have certs, and the webhook is fully configured.

Which goes back to having proper readiness - one of the main goals for 1.0, but not yet in 0.8

Workaround would be to poll for the webhook config and check that it has the cert set.

[kobenauf]

FWIW, I ran across this thread, and I have this issue on Swarm, not using Istio, but using envoy by itself.

[gbaufake]

Facing this issue with Istio 0.8.0 and Bookinfo

[LillyWu]

Facing the same issue with auth enabled on IKS (IBM Kubernetes Service)

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.8-2+af27ab4b096122", GitCommit:"af27ab4b096122049e65b75ee29ac115b1d58f6b", GitTreeState:"clean", BuildDate:"2018-06-14T04:07:19Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

and

istioctl version
Version: 0.8.0
GitRevision: 6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84
User: root@48d5ddfd72da
Hub: docker.io/istio
GolangVersion: go1.10.1
BuildStatus: Clean

[eigokor]

Got same when did cluster nodes recycle.

had to kill ingress gateway pod (which I use to access service)

[liuliqiang]

Same issue in following k8s and istio version:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

and istio:

# istioctl version
Version: 1.0.0
GitRevision: 3a136c90ec5e308f236e0d7ebb5c4c5e405217f4
User: root@71a9470ea93c
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean

[Ghazgkull]

Same issue trying to run through the Bookinfo sample on EKS. The /productpage route was working fine until I applied the bookinfo sample virtualservices.

Following the instructions here: https://istio.io/docs/tasks/traffic-management/request-routing/

Applying the virtual services with: kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

[purpletech77]

Yes, there is a timing issue. (hint from salrashid123 comment)

[moshe0076]

Hi,

Having the same problem with Istio 1.0.2 on eks.2.

Is there any solution or fix planned for this? It basically makes the use of Istio very problematic now.

Thanks!

[th19E]

Same issue trying to run through the Bookinfo sample on EKS. The /productpage route was working fine until I applied the bookinfo sample virtualservices.

Following the instructions here: https://istio.io/docs/tasks/traffic-management/request-routing/

Applying the virtual services with: kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

You need to run the following before applying the virtual service:

kubectl apply -f samples/bookinfo/networking/destination-rule-all-mtls.yaml

[wolfuser99]

Same issue here on local getting:

upstream connect error or disconnect/reset before headers

with

$ istioctl version
Version: 1.0.3
GitRevision: a44d4c8bcb427db16ca4a439adfbd8d9361b8ed3
User: root@0ead81bba27d
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Cleann

and

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

when i execute the httpbin example(this one https://raw.githubusercontent.com/istio/istio/release-1.0/samples/httpbin/httpbin.yaml) with and without the sidecar injected i get about 50% of the request, according to grafana, with code 503 and the messege upstream connect error or disconnect/reset before headers .