search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.31k stars 21.48k forks source link

Connection termination error for ssl URL #44379

Closed Bhanu88 closed 4 years ago

Bhanu88 commented 4 years ago

when we have installed istio 1.4.0 with AKS and trying to use SSL url it is giving following issue

Upstream connect error or disconnect/reset before headers. reset reason: connection termination

Note : same setup is working fine with Istio 1.3.2

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

Karishma-Tiwari-MSFT commented 4 years ago

Thanks for the feedback! We are currently investigating and will update you shortly.

Karishma-Tiwari-MSFT commented 4 years ago

@MicrosoftDocs/aks-pm Can any of you please share insights on what could be the reason behind the error. I can't seem to find any information on this. Thanks :)

Karishma-Tiwari-MSFT commented 4 years ago

@paulbouwer Do you have any insights into this issue the customer is facing? Thanks :)

paulbouwer commented 4 years ago

@Bhanu88 - please can you provide more details about your setup? How have you configured TLS on your ingress?

Bhanu88 commented 4 years ago

Please find the steps below 1.i bought a wild card certificate from azure and from that certificate i extracted cert and key file . 2.Using following command i created the secret in AKS

kubectl create -n istio-system secret tls istio-ingressgateway-certs --key /root/mbv.key --cert /root/mbv.crt

  1. configured the api gateway with following Yaml 
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-app-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - jpr-ser.mbv.com
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - jpr-ser.mbv.com
  1. attached one of the virtual service using following script 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ .Values.appname }}-app
  namespace: istio-system
spec:
  hosts:
  - "{{ .Values.virtualservice.dnshosts }}"
  gateways:
  - {{ .Values.virtualservice.gateways }}
  http:
  - match:
    - uri:
        prefix: /{{ .Values.virtualservice.urlprefix }}
    route:
    - destination:
        host: {{ .Values.appname }}-app.{{ .Values.namespaces }}.svc.cluster.local
        port:
          number: {{ .Values.virtualservice.port }}

values.yaml

replicaCount: 1
appname : purposebk
appversion : 1.0
namespaces : default
envtype : dev

image:
  repository: xyz.azurecr.io/purposebook
  tag: 1768
  pullPolicy: IfNotPresent
  targetport: 8925
  pullsec: xyzz
  ENVVAR1: ASPNETCORE_ENVIRONMENT
  ENVVAR1VAL: Development
  ENVVAR2: KUBE_CLUSTER
  ENVVAR2VAL: DEVOPS

service:
  type: ClusterIP
  port: 80
  targetport: 8925

virtualservice:
  dnshosts: jpr-ser.mbv.com
  port: 80
  gateways: api-app-gateway
  urlprefix: purpose-book
Bhanu88 commented 4 years ago

@paulbouwer when i did the same setup with Version 1.3.2 of istio its working fine ..i also browsed a bit on internet some other guys are also facing the same issue with version 1.4.0 .... and everyone is telling that if we downgrade to version 1.3.2 it works fine

Bhanu88 commented 4 years ago

@paulbouwer yes i configured the TSL at ingress level

Bhanu88 commented 4 years ago

https://discuss.istio.io/t/upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-termination/4434

paulbouwer commented 4 years ago

@Bhanu88 - I found the following in the Istio Discuss thread you reference interesting.

It looks like this is a bug in Envoy that the Istio folks are aware of.

It’s a bug in Envoy. It doesn’t support/screws up Application-Layer Protocol Negotiation(ALPN) If you have a http1.1 service in the backend and a client sending Upgrade: h2c - Envoy will upgrade the connection to http2 and then force the backend to use http2 even if its not supported.

Also, it looks like there is a workaround if you use named service ports.

Check by changing the port name of the jaeger service from ‘query-http’ to ‘http-query’. It seems that from istio 1.4, when the naming convention is not followed it uses the same protocol as in input so http2 if you use https to access the gateway. When specifying http-query, envoy will know that the backend it http1.1 and not http2 and will sned the request accordingly.

Using named service ports is something I mention in the Note in Use intelligent routing and canary releases with Istio in Azure Kubernetes Service (AKS) > Deploy the application.

Could you look at if leveraging named service ports would allow you to continue using Istio 1.4 in your current setup?

Named Service Ports

Example of Named Service Ports

Bhanu88 commented 4 years ago

thanks for the suggestion but as suggested i dont want to change configuration of any component which is not mine and understating my problem but i will not upgrade to istio 1.4 until issue is fixed . I request Microsoft to check for these common scenarios if possible before updating documentation .

Please you can let me know if the issue fixed in future version.

Also i cant find the the link of istio 1.3.2 with AKS documentation anymore Please let me know where i can find it ....

paulbouwer commented 4 years ago

Istio requires that your workloads satisfy a number of requirements - one of these requirements is around named service ports:

Named service ports: Service ports must be named. The port name key/value pairs must have the following syntax: name: <protocol>[-<suffix>].

It has been my experience that following these requirements from the Istio team will ensure the best operating experience.

The official Istio documentation and the discussion site are still the best places to get solutions to broader Istio issues. You can find the installation guides for older Istio versions on the Istio archives site.

Hope that helps.

Bhanu88 commented 4 years ago

Thanks Paul i am closing the issue