HTTP/2 503 upstream connect error or disconnect/reset before headers,two service cannot communcation.

[sandylss]

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?:

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version : 0.7.1
kubectl version :Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:29:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:21:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ?

I install the stable istio.yaml. What happened:

I followed https://istio.io/docs/setup/kubernetes/quick-start.html install the istio,and test the bookinfo works well. Then I deploy two service,server and client,server contains a hiredis database.First, the Client and Server establish a TCP connection.Then the Client encapsulates the HTTP body & header,using http/2 protocol. After that the Client sends a request,the Server decodes the data, it goes to the Hiredis database to perform the corresponding operation, and then returns a response. The server cannot receive the request,and client receive HTTP/2 503 upstream connect error or disconnect/reset before headers But I see from server and client istio-proxy the following logs: server "PUT /nrf-nfm/v1/nf-instances/test HTTP/2" 503 UF 243 57 2 - "-" "-" "eb2980b4-4b9a-9f11-9b8f-bd470d58945d" "server:6380" "127.0.0.1:6380" client [2018-05-29T08:18:43.582Z] "PUT /nrf-nfm/v1/nf-instances/test HTTP/2" 503 - 243 57 4 3 "-" "-" "eb2980b4-4b9a-9f11-9b8f-bd470d58945d" "server:6380" "10.244.0.78:6380"

What you expected to happen:

server and client can work well. How to reproduce it: The yaml file is

---
apiVersion: v1
kind: Service
metadata:
  name: server
  labels:
    app: server
spec:
  ports:
  - port: 6380
    name: http2
    targetPort: 6380
  selector:
     app: server

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: server
 #       version: v1
    spec:
      restartPolicy: Always
      containers:
      - name: server
        image: nrf:server
        imagePullPolicy: IfNotPresent
#        resources:
#          requests:
#            cpu: 100m
#            memory: 100Mi
        ports:
           - containerPort: 6380
        stdin: true
        tty: true
#        command: ['nohup redis-server > /dev/null &']

---

apiVersion: v1
kind: Service
metadata:
  name: client
  labels:
    app: client
spec:
  ports:
  - port: 6380
    name: http2
    targetPort: 6380
  selector:
     app: client
---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: client
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: client
#        version: v1
    spec:
      containers:
      - name: client
        image: nrf:k8scli-2
        imagePullPolicy: IfNotPresent
        env:
        - name: GET_HOSTS_FROM
          value: dns
        ports:
        - containerPort: 6380
        stdin: true
        tty: true

Feature Request: N

Describe the feature:

[johnzheng1975]

Same for https://github.com/istio/issues/issues/366

[sandylss]

@johnzheng1975 sorry，I confirm the pod ip in the istio-proxy and app service is same,but ,it still return the error

HTTP/2 503
upstream connect error or disconnect/reset before headers

I notice the server istio-proxy receive the request,and the source.uid is the 127.0.0.1:6380 ,the port is also be listened tcp 0 0 server-6d9d4d99df-:6380 *:* LISTEN what if I miss something ?