search

istio / istio

Connect, secure, control, and observe services.
https://istio.io
Apache License 2.0
35.95k stars 7.76k forks source link

Is it possible to disable http2 and always use http 1.1? #14708

Closed tjholwik closed 5 years ago

tjholwik commented 5 years ago

we are using istio 1.1.7 and we have an issue with http2, is there a way to configure istio to use only http 1.1 protocol by default ? didnt find any way in the docs how to configure it...

howardjohn commented 5 years ago

Are your ports named http-.. or http2-...? See https://istio.io/docs/setup/kubernetes/prepare/requirements/ "Named service ports" section

tjholwik commented 5 years ago

@howardjohn - I've add another port for http like following and still we are not able to use http instead of http2, any idea? 

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: wgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https-w
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: bide-tls
      hosts:
        - istio.dev1.stv.net.tsr
    - port:
        number: 80
        name: http-wd
        protocol: HTTP
      hosts:
        - istio.dev1.stv.net.tsr
      tls:
        httpsRedirect: true
howardjohn commented 5 years ago

What issues are you having with http2? Where do you have issues and where do you want to disable http2?

There are 3 different connections:

CLIENT ----- 1 -----> CLIENT SIDECAR ------ 2 ----> SERVER SIDECAR ----- 3 ----- > SERVER

Where is the problem and where do you want no http2?

tjholwik commented 5 years ago

@howardjohn - we are not using side cars yet, we are testing istio as replacement for nginx hence we want that all the incoming traffic via will be with http and not http2 , how we can do it ? (this is something that we were able to configure with nginx ingress) we have tools that needs to connnects to our pods and support only http1.1 ...

rshriram commented 5 years ago

Envoy has a default auto codec for inbound listeners. It detects and upgrades to http2 if the client supports it. If not it falls back to http1. If you are seeing problems with this pattern, I suggest you report this to the envoy folks. They should be able to fix it quickly and we can pull the updated envoy.

tjholwik commented 5 years ago

@rshriram - thanks, I've post it now.

lizan commented 5 years ago

@tjholwik IIUC, you want envoy only serve HTTP/1.1, regardless what client support? If so this is not an envoy bug or issue, it is pilot always generating HCM with codec auto. cc @rshriram

tjholwik commented 5 years ago

How to proceed ?

johnjjung commented 5 years ago

https://istio.io/docs/ops/troubleshooting/network-issues/#404-errors-occur-when-multiple-gateways-configured-with-same-tls-certificate

We're having issues with HTTP2 as well

Not sure why but there are 404's happening for all endpoints but one only in modern browsers that use http2 (we think). We tried using a single wildcard gateway but this does not seem to work, all endpoints go down.

Is there a way to temporarily downgrade all http2 requests? curl seems to work (which I assume is not http2)

FridayPush commented 5 years ago

@johnjjung I'm hitting this same issue which for me appears to be #9429; The browsers are over aggressively using open connections. Trying to find a work around.

johnjjung commented 5 years ago

@johnjjung I'm hitting this same issue which for me appears to be #9429; The browsers are over aggressively using open connections. Trying to find a work around.

Yea, we had traditional LB for services in parallel and our uptime checks had LB's up but Istio Gateways/Virtual Services down, and ended reverting back to GCP-load balancers for our services. It's strange the latency from our uptime checks went down significantly as well. We'll be experimenting on our sandbox-cluster:

johnjjung commented 5 years ago

@FridayPush any work around discovered yet?

istio-policy-bot commented 5 years ago

🚧 This issue or pull request has been closed due to not having had activity in the last 105 days. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

eddiewang commented 4 years ago

having problems with envoy not respecting the server's inability to serve http2. instead the connection gets terminated.

rjshrjndrn commented 4 years ago

Facing the same Issue. Is there any annotation which you can use to tell don't use http2 for a particular deployment ?

Deathhush commented 4 years ago

I am have encountered a similar issue and not sure if this is the right place to discuss this. I have a micro service which only supports http 1.1. Is it possible for the gateway to support 2.0 while still forward the request to a 1.1 only backend service?

delmendo commented 4 years ago

I'm using a nodejs/express app. They don't currently support http2 in expressjs.

I'm doing SSL termination at the ingress. Since it is behind SSL chrome and Envoy announces support for both h2 and http/1.1 - the chrome browser submits requests as http/2.0. Chrome -> http/2.0 -> Envoy -> Node Service

There doesn't seem to be a way to prevent the ingress from announcing a protocol the backend doesn't support.

To work around it I had to add an nginx proxy in the middle (which also has some issues because nginx can't do http/1.1 and http/2.0 on the same port when SSL isn't present ) I'd love to be able to control the ALPN Protocols from the TLS_CONTEXT - right now it looks like this is hard coded in the pilot backend.

It looks like it is controlled by this constant pkg/networking/util/util.go:var ALPNHttp = []string{"h2", "http/1.1"}

And then it is used here: pkg/networking/core/v1alpha3/gateway.go on line 425

I don't know go well enough to build up the infrastructure to make this controlled by the gateway defenition instead of hard coding it.

klarose commented 4 years ago

@rshriram Is there a way this could be reopened? There is clearly interest in it, and there has been since it was initially closed as stale.

We are running into a similar issue where a bug in envoy's ext_authz filter prevents it from forwarding bodies when the downstream is http 2. I'd like to force downstream connections to http 1.1 until we get the fix. However, I cannot do this. I'm sure similar issues will occur in the future, so being able to control the version used is pretty important.

jbartlet commented 4 years ago

Not sure if this helps, but I ran into the same issue as @delmendo and it was solved by modifying the Service definition for the app (not the Istio virtualservice, but the K8S service). Our service manifests have no 'name' field in the spec section. If they don't, Envoy simply sends in what the client browser negotiated, H2 in my case. However, if you ensure there is a name field and populate it with a protocol from https://istio.io/docs/ops/configuration/traffic-management/protocol-selection/ then Envoy will adjust as required. Putting 'name: http' in the service manifest resulted in the client still negotiating H2, but Envoy only sending H1.1 to the backend, which works.

Darwiner commented 4 years ago

I haven't had the chance/time to test it out yet, but has anyone tried to enable

--set values.pilot.enableProtocolSniffingForInbound=true

...with Istio 1.4, to see if it "fixes" this, without having the need to go and assign a protocol name for each and every service definition that Istio needs to hit?

One difference seems to be that 1.3 has Protocol Sniffing on by default (https://archive.istio.io/v1.3/docs/ops/traffic-management/protocol-selection/), while 1.4 (for incoming traffic), is off...

rjshrjndrn commented 4 years ago

in istio 1.5(atleast that's what I saw) there is an option in Destination rule setting, h2UpgradePolicy: DO_NOT_UPGRADE https://istio.io/docs/reference/config/networking/destination-rule/#ConnectionPoolSettings-HTTPSettings

vivekpd15 commented 4 years ago

Facing the same issue in Istio 1.5

h2UpgradePolicy: DO_NOT_UPGRADE didn't work for me. Neither did values.pilot.enableProtocolSniffingForInbound=true.

Anyone has a solution to this yet?

rjshrjndrn commented 4 years ago

what's the exact error are you facing ?

vivekpd15 commented 4 years ago

@rjshrjndrn I have a website deployed behind nginx web server. I am running Istio 1.5 with mTLS and SDS enabled using the sidecar pattern along with cert-manager for managing SSL certificates. I have the gateway and virtual service for the website created and I can access it on plain HTTP port but when I want to access it on HTTPS, it gives me a 404 instead. The TLS handshake works well so I am sure it's not SSL. In the curl logs I can see that the connection was upgraded to http2 in envoy. Nginx is using normal http protocol without SSL.

❯ curl -I -Lv https://example.com

Similar issue: https://github.com/istio/istio/issues/13578

rjshrjndrn commented 4 years ago

weird. If it's working on 80, then it should work on 443 also. I think you have to reverify your gateway config. kubectl edit gateways.networking.istio.io

ragingpastry commented 3 years ago

We ran into this issue and ended up disabling HTTP/2 in istio with the following config.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: disable-alpn-h2
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: FILTER_CHAIN
    match:
      listener:
        filterChain:
          sni: "*.mygateway.com"
    patch:
      operation: MERGE
      value:
        transportSocket:
          name: envoy.transport_sockets.tls
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            commonTlsContext:
              alpnProtocols:
                - "http/1.1"
              tlsCertificateSdsSecretConfigs:
                - name: kubernetes://wildcard-cert
                  sdsConfig:
                    ads: {}
                    resourceApiVersion: V3
Darnesey commented 3 years ago

We ran into this issue and ended up disabling HTTP/2 in istio with the following config.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: disable-alpn-h2
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: FILTER_CHAIN
    match:
      listener:
        filterChain:
          sni: "*.mygateway.com"
    patch:
      operation: MERGE
      value:
        transportSocket:
          name: envoy.transport_sockets.tls
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            commonTlsContext:
              alpnProtocols:
                - "http/1.1"
              tlsCertificateSdsSecretConfigs:
                - name: kubernetes://wildcard-cert
                  sdsConfig:
                    ads: {}
                    resourceApiVersion: V3

Facing similar issues with Istio 1.11.3. Neither option worked for me h2UpgradePolicy: DO_NOT_UPGRADE values.pilot.enableProtocolSniffingForInbound=true

@ragingpastry's EnvoyFilter only appears to work if you don't already have TLS configurations set. If your configurations are already set, then the most you achieve is appending another http/1.1 value inside the alpnProtocols list as apparently is the algorithm for proto merge semantics. 

"alpnProtocols": [
  "h2",
  "http/1.1",
  "http/1.1"
]

I managed to get @ragingpastry's config to work so long as I removed tls configurations from my gateway ahead of time and used an EnvoyFilter config to manually MERGE the protocol requirements and the tls configurations into the filter chain.

This is a very awkward solution though that feels like I'm building an ikea furniture without the instructions and being left with extra screws.

nmnellis commented 2 years ago

for anyone else looking for the fix to this, this is what i used. note the gateway using http protocol

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: http
    hosts:
    - frontend.localhost
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: disable-ingress-h2
  namespace: istio-system
  labels:
    istio: ingressgateway
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        portNumber: 8443
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
### Set the codec type to http1
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP1
# Add the SNI
  - applyTo: FILTER_CHAIN
    match:
      context: GATEWAY
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        filterChainMatch:
          serverNames:
          - frontend.localhost
### Enable HTTPS for only http/1.1 using out kuberentes secret for certs
  - applyTo: FILTER_CHAIN
    match:
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        transportSocket:
          name: envoy.transport_sockets.tls
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            commonTlsContext:
              tlsParams:
                tlsMaximumProtocolVersion: TLSv1_3
                tlsMinimumProtocolVersion: TLSv1_2
              alpnProtocols:
              - http/1.1
              tlsCertificateSdsSecretConfigs:
              - name: kubernetes://gateway-certs # your kubernetes secret name here
                sdsConfig:
                  ads: {}
                  resourceApiVersion: V3
            requireClientCertificate: false
### Add back the TLS inspector for the SNI
  - applyTo: LISTENER
    match:
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        listenerFilters:
        - name: envoy.filters.listener.tls_inspector
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
  workloadSelector:
    labels:
      istio: ingressgateway
feiyao commented 2 years ago

Ideally, it should be able to REPLACE "transportSocket.alpnProtocols" within whatever values you want but the issue seems to be that Istio "operation: REPLACE" only applies to HTTP_FILTER, and NETWORK_FILTER according to the latest document. And it doesn't seem to work, not sure why.

marcozov commented 2 years ago

The only way I found to make this work was to rewrite the entire filter chain through EnvoyFilter. What I believe could help is:

ningyougang commented 1 year ago

This is my latest research work: https://github.com/istio/istio/discussions/44925

In short words