k-wall
commented
4 years ago In the failing case, the client application receives a HTTP 1.1 200 response code in response to its HTTP/1.1 websocket upgrade request rather than the expected HTTP/1.1 101 Switching Protocols.
Turning on logging on the haproxy (output below), I see that the happroxy appears to be 'bridging' the request from HTTP 1.1 to HTTP 2 when communicating with the backend (AMQ Interconnect - which is HTTP2 enabled), but fails to bridge the response on the return path. It turns a HTTP/2 200 response into an HTTP/1.1 200 response rather than the expected HTTP/1.1 101 Switching Protocols. This causes the end user's application to fail.
Failing - OpenShift 4.4.5 haproxy (-d flag).
00000007:fe_sni.clireq[0019:ffffffff]: GET / HTTP/1.1
00000007:fe_sni.clihdr[0019:ffffffff]: host:
messaging-wss-queuespace-enmasse-infra.apps.maas-ocp-aws.openshift-aws.rhocf-dev.com:443
00000007:fe_sni.clihdr[0019:ffffffff]: upgrade: websocket
00000007:fe_sni.clihdr[0019:ffffffff]: connection: upgrade
00000007:fe_sni.clihdr[0019:ffffffff]: sec-websocket-key:
m2yOIRksdcZv+jQ28TIgoA==
00000007:fe_sni.clihdr[0019:ffffffff]: origin:
http://messaging-wss-queuespace-enmasse-infra.apps.maas-ocp-aws.openshift-aws.rhocf-dev.com:443
00000007:fe_sni.clihdr[0019:ffffffff]: sec-websocket-protocol: amqp
00000007:fe_sni.clihdr[0019:ffffffff]: sec-websocket-version: 13
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvrep[0019:001a]:
HTTP/2.0 200
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0019:001a]:
server: qpid-dispatch-router
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0019:001a]:
content-type: text/html
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0019:001a]:
accept-ranges: bytes
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0019:001a]:
cache-control: no-store
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0019:001a]:
etag: 000008805EBD6156
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.srvcls[0019:001a]
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.clicls[0019:001a]
00000007:be_secure:enmasse-infra:messaging-wss-queuespace.closed[0019:001a]For comparison - Succeeding - OpenShift 4.3.10 haproxy (-d flag).
00000002:public_ssl.accept(0007)=000c from [192.168.64.1:63877] ALPN=<none>
00000003:fe_sni.accept(0008)=0010 from [192.168.64.1:63877] ALPN=<none>
00000003:fe_sni.clireq[0010:ffffffff]: GET / HTTP/1.1
00000003:fe_sni.clihdr[0010:ffffffff]: host:
messaging-wss-queuespace-enmasse-infra.apps-crc.testing:443
00000003:fe_sni.clihdr[0010:ffffffff]: upgrade: websocket
00000003:fe_sni.clihdr[0010:ffffffff]: connection: upgrade
00000003:fe_sni.clihdr[0010:ffffffff]: sec-websocket-key:
KxnlHDdWtEDs9C36fhAJBQ==
00000003:fe_sni.clihdr[0010:ffffffff]: origin:
http://messaging-wss-queuespace-enmasse-infra.apps-crc.testing:443
00000003:fe_sni.clihdr[0010:ffffffff]: sec-websocket-protocol: amqp
00000003:fe_sni.clihdr[0010:ffffffff]: sec-websocket-version: 13
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvrep[0010:0011]:
HTTP/1.1 101 Switching Protocols
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0010:0011]:
Upgrade: WebSocket
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0010:0011]:
Connection: Upgrade
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0010:0011]:
Sec-WebSocket-Accept: LFXNtQiJmqvAfaMaQWRKqFDIMfo=
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvhdr[0010:0011]:
Sec-WebSocket-Protocol: amqp
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.srvcls[0010:adfd]
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.clicls[0010:adfd]
00000003:be_secure:enmasse-infra:messaging-wss-queuespace.closed[0010:adfd]
Describe the bug
A defect in a new feature delivered on the OpenShift 4.4 release line causes a regression for certain AMQP websocket use-cases that utilise a Router with with a TLS termination option
reencrypt. Any attempted websocket connections will fail with the newer OpenShift versions.A workaround is to use a different OpenShift Route TLS termination type (passthrough, edge).
To Reproduce Steps to reproduce the behavior:
Expected behavior Test should pass.