reverse proxy WebSocket upgrade request results in a 400 bad request (Supermicro IPMI)

bugtoo commented 4 years ago

Hello everybody! I am building a reverse proxy configuration with Caddy 2.2.0 to expose a Supermicro IPMI to another network. For those unfamiliar with IPMI, it's basically a web interface of a bare metal server, providing, among other things, remote access to Keyboard ,Video and Mouse of the server. This works with a websocket session with noVNC.

I can make the reverse proxy for the http part to work, but as soon as I start the websocket session, I got a 400 Bad Request from the IPMI server. I tried with different Origin/Host combination and with different Headers manipulation, and I also captured a PCAP of the session: even if the upgrade request is identical compared to another going directly to the IPMI, I always get a 400 Bad Request when going through the proxy. I am running this in a container, so I quickly switched to nginx to test it and there it works out of the box. I believe there's something low level involved that I can't see with a packet capture.,or something obvious that I don't see.

Here's Caddy config:

{
  http_port  80
}
server-1.ipmi.myservers.cloud:80 {
    reverse_proxy http://192.168.1.33
}

Here's nginx config:

events {}
http {
  server {
    listen 80;
    server_name server-1.ipmi.myservers.cloud; 

    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $host;
      proxy_pass http://192.168.1.33;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
    }
  }
}

From my understanding, the two configurations are equivalent, and I can see it also in the pcap capture that the outcome is similar.

Any suggestions? How can I help in acquiring more debug? There's nothing appearing in the logs.

Here's failed session with caddy:

`GET / HTTP/1.1
Host: server-1.ipmi.myservers.cloud
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Cache-Control: no-cache
Connection: Upgrade
Cookie: langSetFlag=0; language=English; SID=bApuabahRMaWt4f; mainpage=remote; subpage=man_ikvm_html5
Dnt: 1
Origin: http://server-1.ipmi.myservers.cloud
Pragma: no-cache
Sec-Websocket-Extensions: permessage-deflate
Sec-Websocket-Key: cQREl1VmamJgiM+ww/OU5Q==
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Proto: http

HTTP/1.1 400 Bad Request
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-XSS-Protection: 1; mode=block
Content-Type: text/html
Content-Length: 349
Date: Mon, 05 Oct 2020 13:04:09 GMT

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>`

Here's successfull session with nginx:

GET / HTTP/1.1
X-Forwarded-For: xxx.xxx.xxx.xxx
Host: server-1.ipmi.myservers.cloud
Upgrade: websocket
Connection: upgrade
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0
Accept: */*
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Sec-WebSocket-Version: 13
Origin: http://server-1.ipmi.myservers.cloud
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: rVMWkkhQoJ/E+A3/eZYGjA==
DNT: 1
Cookie: langSetFlag=0; language=English; SID=NP12ZnzPZ6Z4R2U; mainpage=remote; subpage=man_ikvm_html5
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: uwjiwVYTrWqXA4Jj8wx10iRtLHs=

..RFB 055.008
....'3..e........9......w+..g......@q...........\.........rC.A#..:..._C-.HB..{F/.P#~..rC..rC..rC..rC..rC..rC.........5.%G4.4.... ...................ATEN iKVM Server.....o.P.................Q.N....;.).~.  9........524 ADMIN 192.168.144.249........................................................................................................................................................................................................................................~.................W.......|....%.8._ .....o.w.k......7..t{.jk....V:kK...&hmK6....2...Px.v}*.`.....k.7.....dp....m...6......V.Wsk....e.$`.a[.z....9.n.P..i`..?........<.....Z..9
.....=.a...=..e.4..A.Y...U.j...S%.........4...<.........8..X|.......-.......b>cm....x....g.}]9.........`.n..]27.....85Z...1.c....2.....n...g.'..........5Zh^>7...FU....W...q......i...-c........X..F=.....;-..B%{.<....r,1u;.|;........{..~....k.....~y..l.....<.h..p.dX|/ZVy../o

The only obvious thing is the order of the headers which are different: in nginx we get the "Upgrade" and "Connection" headers much earlier compared to caddy: it keeps the original order of the request coming from the browser, but it shouldn't matter.

Thanks!

Daniel

francislavoie commented 4 years ago

Thanks for the detailed report.

That all looks okay to me as well. That error is clearly coming from IPMI, so we'll need more information as to why it's rejecting the request.

bugtoo commented 4 years ago

Hi @francislavoie , it looks like there's no way to get more informations on the IPMI itself as it is a closed system - I wish Supermicro could give us some insights but they closed the ticket saying that the architecture is not supported. Is there a way to force Caddy to reorder the headers so that "Upgrade" and "Connection" come at the same level as with nginx?

francislavoie commented 4 years ago

I'm 99% sure header order is not the problem. Headers are a key-value mapping, not an ordered set. There's no way for that to change, because Caddy uses golang's map type under the hood, which is an unordered set.

I notice that Caddy adds X-Forwarded-Proto while nginx doesn't. I guess there's a possibility that breaks things, but that would surprise me. You should be able to suppress that by adding header_up -X-Forwarded-Proto to your reverse_proxy config. https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#headers

bugtoo commented 4 years ago

I already tested all the possible combinations (stripping headers, adding headers), nothing changes. I gave it another try:

GET / HTTP/1.1
Host: server-1.ipmi.myservers.cloud
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Cache-Control: no-cache
Connection: Upgrade
Cookie: langSetFlag=0; language=English; SID=j5ggH6NXxct69FC; mainpage=remote; subpage=man_ikvm_html5
Dnt: 1
Origin: http://server-1.ipmi.myservers.cloud
Pragma: no-cache
Sec-Websocket-Extensions: permessage-deflate
Sec-Websocket-Key: MEL6GPcPFPrKYXAAnEuilw==
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: xxx.xxx.xxx.xxx

HTTP/1.1 400 Bad Request
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-XSS-Protection: 1; mode=block
Content-Type: text/html
Content-Length: 349
Date: Mon, 05 Oct 2020 15:35:25 GMT

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

No luck :(

It must be lower level then, not sure how to catch the differences between the two requests though.

francislavoie commented 4 years ago

Alright - well I'll close this, I don't think there's anything actionable on our end. There's not really enough information for us to do anything with at the moment. I hope you can convince Supermicro to tell you why a 400 might be returned so we can investigate further. We know that websockets works fine for others.

mholt commented 4 years ago

I'm 99% sure header order is not the problem.

It shouldn't be, but you never know what kind of awful enterprisey apps make stupid, spec-violating assumptions. :sweat: I definitely wouldn't be surprised if they expect a certain order to the headers and if that's breaking things.

This looks like something Supermicro will have to fix.

akosyakov commented 3 years ago

Caddy writes Websocket, but Nginx writes WebSocket.

francislavoie commented 3 years ago

@akosyakov this is probably not the right place to ask a question. This is a long-since closed issue.

Please ask your usage questions on the Caddy community forums. We prefer to keep the GitHub issue board for bugs and feature requests. Don't forget to fill out the thread template so we can help you!

caddyserver / caddy

reverse proxy WebSocket upgrade request results in a 400 bad request (Supermicro IPMI) #3778