SPDY is deprecated. Switch to HTTP/2.

[lavalamp]

Filing for tracking purposes.

See #7392 where the spdy package we use doesn't even exist any more. (And btw it has bugs anyway, possibly security bugs-- see the trophies here: https://github.com/dvyukov/go-fuzz)

Blocked on github.com/docker/spdystream switching to HTTP/2.

(Note: HTTP/2 is based on SPDY, with just a few differences-- hopefully this transition will be easy.)

@thockin @ncdc

[ncdc]

Waiting on HTTP/2 support for Go that includes low level client/server ability to create and handle streams (https://github.com/bradfitz/http2).

https://github.com/docker/spdystream/pull/53 pulls the upstream spdy code into spdystream.

[lavalamp]

Ah, after docker/spdystream#53 is merged, we can at least make godep happy again. :)

[ncdc]

@lavalamp spdystream PR is merged, FYI

[ncdc]

@thockin @lavalamp it looks like the Go HTTP/2 issue was just closed (https://github.com/golang/go/issues/6891#issuecomment-161070230). I glanced quickly through the code, but I don't see a way to hijack the connection and create our own streams, like we do with spdystream. Hopefully we can get this functionality in there at some point...

[ncdc]

cc @smarterclayton

[ncdc]

I created https://github.com/golang/go/issues/13444 as an RFE for end-user stream support

[ddysher]

What's the plan for the deprecation? We are using nginx in front of apiserver which drops support for SPDY since version 1.9.5. Our workaround is to use an older nginx, but would like to see http/2 support go upstream :)

[ncdc]

I prototyped this with a prerelease version of go1.6. It worked somewhat, but there were some rough spots (probably stuff I was doing incorrectly). We can't move to http/2 until go1.6 at the earliest, and even then we'll have to do R&D to see if it's technically feasible. I had been hoping that the http/2 support in go1.6 would allow me to create streams just like we can do in spdystream, but that isn't exposed via the API. For my prototype, I ended up writing a simple multiplexer just like @smarterclayton wrote for the websocket support we have in Kubernetes now.

[ddysher]

Thanks for the heads up. Looking at the linked comments, there is no intention to add the support in go1.6, so I guess it's probably still months away.

[JensRantil]

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue now? Anything else stopping us?

[smarterclayton]

A usable HTTP/2 framer in Go that allows us hijack control and the ability to stand up the connection. Last I saw we were still blocked without explicitly forking the Go implementation.

In the meantime, websockets continue to be an option (with portforward being the last bit remaining).

On Oct 29, 2016, at 2:22 PM, Jens Rantil notifications@github.com wrote:

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue now? Anything else stopping us?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/7452#issuecomment-257107405, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p-fe3BrfKe4VjAQyYLOgK7AjvhLkks5q447ngaJpZM4EK7sh .

[ncdc]

We should be able to use HTTP/2 without any changes to the Go implementation. But we would have to write our own code to do multiplexing of multiple data streams over a single request/response body pair, which we'd rather not do if we can avoid it.

[bradfitz]

@ncdc, if it helps, https://godoc.org/golang.org/x/build/revdial is used by the Go build system to multiplex multiple net.Conns and a net.Listener all over a single net.Conn.

[fraenkel]

@bradfitz Thanks for the pointer, but it doesn't fix the underlying issue. While SPDY is being used, it is not following HTTP request/response semantics. A logical connection is created and then data is sent in either direction as needed. It would be equivalent to the client sending an HTTP request and the server doing a Push whenever.

[ncdc]

@fraenkel I don't think there's any technical limitation in the facilities provided by the Go HTTP/2 implementation. As I wrote above, we would just need to write our own muxing/framing code, which we'd rather avoid (or find a library that can do it?).

[fraenkel]

@ncdc It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done.

[smarterclayton]

There's definitely benefit to implementing a gRPC endpoint vs continuing to have to maintain a franken-protocol. That is tied up with the CRI changes though.

On Mon, Nov 28, 2016 at 11:25 AM, Michael Fraenkel <notifications@github.com

wrote:

@ncdc https://github.com/ncdc It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/7452#issuecomment-263317577, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p4YovgyT-NPE9MilgvO0kf_CaSksks5rCwB4gaJpZM4EK7sh .

[luxas]

Ping, any update on this?

[wlan0]

We at Rancher Labs are also interested in this change.

[smarterclayton]

We can actually just move to web sockets in the near term. The server has supported it for a while and HOL blocking is not an issue

[smarterclayton]

Tracking #48633 to move kubectl to web sockets.

[smarterclayton]

General proposal:

Mark SPDY deprecated in 1.8. Clients (https://github.com/kubernetes/kubernetes/issues/48633) switch to using websockets in 1.8/1.9/1.10. We stop adding new streaming support for SPDY.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

[luxas]

/remove-lifecycle stale /lifecycle frozen

Assuming @smarterclayton is still working on https://github.com/kubernetes/kubernetes/pull/50428

[josdotso]

Is there any status update on this issue?

[mfranczy]

Please ignore my reference, it has nothing to do with that issue, if I could I would remove it, sorry.

[ensonic]

We run a http/2 reverse proxy to expose a on-prem api-server. This issues breaks kubectl exec and thereby also tools like helm.

[dims]

long-term-issue (note-to-self)

[mikedanese]

I've started working on streaming over h2 here:

https://github.com/kubernetes/kubernetes/compare/master...mikedanese:h2?expand=1

I hope to make some progress in 1.14

[mikedanese]

I'd like to see this happen in two parts:

1. support all streaming codecs over http/2.
2. deprecate spdy streaming codecs.

I think (1) is probably the biggest win.

[mikedanese]

I have a rough prototype of (1) working e2e from kubectl to CRI. Migration will be tricky. I haven't thought of a way for clients to discover h2 streaming capabilities since the entire stack (kubectl -> apiserver -> kubelet -> cri) needs to support streaming over h2. With a significantly more complex implementation, the upgrade aware proxy could translate between clients and servers speaking different protocols. If that were the case, we'd likely only need single hop discovery to transition smoothly.

[mikedanese]

https://github.com/kubernetes/kubernetes/issues/71411 is an example of the dangers of dropping an http connection to TCP. There's a category of bugs to be avoided but this one was particularly insidious.

[scult]

Any update on this?

[mikedanese]

We are blocked on support for https://tools.ietf.org/html/rfc8441 in x/net/http2 if we don't want to implement something non-standard again.

[jayunit100]

Is it possible for the API server to support HTTP2 for watches etc... while still using HTTP1 for everything else? xref https://github.com/kubernetes/client-go/issues/374. Forgive me if this question is a little ignorant, im not up on the latest API stuff :).

[jmillikin-stripe]

Has there been any progress on this since May? The use of SPDY for kubectl exec continues to be a pain point because it's no longer supported by common network proxies (nginx, haproxy, envoy, and so on).

Naively, I would expect something like interactive sessions to be implemented on top of gRPC or another thin layer on the normal HTTP/2 framing. Websockets, CONNECT, and other ways to get a raw byte stream have a difficult security model and need much more support in intermediate software compared to plain HTTP/2.

[jfrabaute]

Hi,

I have a setup where proxying kubectl exec is working, using openresty (docker image openresty/openresty:1.15.8.1-2-stretch to be precise). It should also work with nginx as openresty is based on nginx.

The relevant config in the location block to have kubectl exec proxied correctly is to add:

              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "Upgrade";

this is the listen line in the server block:

listen 443 http2 ssl

The proxied kube-apiserver is running 1.14.3-gke.11 (so, k8s 1.14 with gke modified version).

[brooksmtownsend]

Hello all, any progress on this issue late 2019 / early 2020? We are using an AWS ALB for the rest of our application, but since AWS ALBs do not support the SPDY protocol we have to use a different load balancer solution just for the k8s API.

[elixxx]

Hey @brooksmtownsend i am sure switching to AWS NetworkLoadBalancer is the only solution at the moment. I tried for days to get it running, ALB and ELB don't support SPDY and i can't imagine AWS will add support. So please safe time and use NLB.

[vespian]

Is there any status update on this issue?

[librannk]

any update on this ??

[matiasah]

Hi,

I have a setup where proxying kubectl exec is working, using openresty (docker image openresty/openresty:1.15.8.1-2-stretch to be precise). It should also work with nginx as openresty is based on nginx.

The relevant config in the location block to have kubectl exec proxied correctly is to add:

              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "Upgrade";

this is the listen line in the server block:

listen 443 http2 ssl

The proxied kube-apiserver is running 1.14.3-gke.11 (so, k8s 1.14 with gke modified version).

You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?

[landorg]

You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?

... or kong? as it's based on openresty somehow.

[adisky]

/remove lifecycle-frozen Some work is going on here https://github.com/kubernetes/kubernetes/issues/89163

[wgahnagl]

still relevant! known tech debt but we don't have anyone working on fixing it :(

[mmiranda96]

/kind cleanup

[arianitu]

Just want to mention that nginx does not support HTTP/2 when proxying. Are you guys sure you want to use HTTP/2?

Specifically:

client (http2) -> nginx -> (http2 does not work here) kube-api-server

We are running into essentially the same issue with our API (picking a good bidirectional streaming protocol) and we're trying to pick a good option that works well with proxying servers like nginx. We implemented HTTP/2 initially and then realized, oops, can't use it with nginx so now we are abandoning the implementation.

We are looking at replacing HTTP/2 with HTTP/1 + WebSockets which seem to work well with nginx.

[lavalamp]

Thanks -- at least some of our protocols are already available via websockets and that's a good reason to go all in on that.

Note that there is also an apiserver <-> kubelet / container runtime leg of some calls (exec, attach), and this also needs to not use SPDY any more, and it doesn't need to go through a proxy. So that needn't use websockets (although it could).

[sfxworks]

I believe that would work out of the box with cloudflare

[slandelle]

some calls (exec, attach) ... and it doesn't need to go through a proxy

@lavalamp Why is that?