cleartext HTTP/2 connections

[wtn]

I'm testing out a setup where HA-Proxy version 2.1.4 handles TLS termination and passes requests to backend processes, including Falcon version 0.36.4.

It works when proxied connections use plaintext HTTP/1.1, which is the default in HAProxy. However, proxied connections with plaintext HTTP/2 fail. Here's the log message:

 warn: Falcon::Command::Top
     | Updating Encoding.default_external from US-ASCII to UTF-8
 info: Falcon::Command::Serve
     | Falcon v0.36.4 taking flight! Using Async::Container::Threaded {:count=>8}.
     | - Binding to: #<Falcon::Endpoint http://localhost:9292/ {}>
     | - To terminate: Ctrl-C or kill 30161
     | - To reload configuration: kill -HUP 30161
 info: Async::Container::Thread::Instance
     | - Per-process status: kill -USR1 30161
error: Async::Task
     |   Protocol::HTTP1::InvalidRequest: "SM"
     |   → protocol-http1-0.12.0/lib/protocol/http1/connection.rb:186 in `read_request'
     |     async-http-0.52.0/lib/async/http/protocol/http1/request.rb:31 in `read'
     |     async-http-0.52.0/lib/async/http/protocol/http1/server.rb:40 in `next_request'
     |     async-http-0.52.0/lib/async/http/protocol/http1/server.rb:61 in `each'
     |     async-http-0.52.0/lib/async/http/server.rb:53 in `accept'
     |     async-io-1.29.0/lib/async/io/server.rb:32 in `block in accept_each'
     |     async-io-1.29.0/lib/async/io/socket.rb:73 in `block in accept'
     |     async-1.25.2/lib/async/task.rb:258 in `block in make_fiber'

[wtn]

I realize that falcon host is recommended for production mode. However, the falcon virtual server requires specifying exactly one authority/hostname per rack app, which doesn't work in dynamic/many hostname apps.

[ioquatix]

However, proxied connections with plaintext HTTP/2 fail. Here's the log message:

You are going to have to explain what your setup is trying to achieve. You can see that the request is going to the HTTP/1 protocol:

 |   Protocol::HTTP1::InvalidRequest: "SM"
 |   → protocol-http1-0.12.0/lib/protocol/http1/connection.rb:186 in `read_request'

So, I'm not sure how this is supposed to work.

However, the falcon virtual server requires specifying exactly one authority/hostname per rack app

In the future we will support some internal rewriting of requests.

[wtn]

My interpretation is that the upstream is making a valid cleartext HTTP/2 request that's being incorrectly interpreted as HTTP/1 by falcon. I haven't investigated further to confirm/refute.

I am reporting the issue because it's common to run rack services behind a reverse proxy for a variety of reasons. In my specific case, the reasons are ➀ need to host non-rack HTTPS services on the same hostname and ➁ need to handle requests for arbitrary hostnames in the same app instance.

I appreciate that falcon is trying to incorporate virtual server capabilities in an integrated manner. However, I don't see why it has to be the only "production mode" option. People migrating from puma will be looking for a drop-in replacement that binds to a socket and processes requests without the extra falcon host layer.

[ioquatix]

I will write documentation about how to do what you want. Nothing you've asked for is tricky.

[ioquatix]

I've added a documentation wiki. It needs a lot of work, but I addressed your issue under deployment. Do you mind trying it out?

https://github.com/socketry/falcon/tree/master/wiki

To run the wiki: bundle install && falcon serve. If you see any issues or make any edits, feel free to submit a PR.

[gottlike]

I kind of had the same issue and am currently using Puma for that reason (it seems to be much easier to set up behind a reverse proxy). Maybe I'm too inexperienced with regard to rack/ruby webserver internals, but I don't "get" the wiki entry at all :sweat_smile:.

If I have a Roda application as a config.ru file like this:

require 'bundler'
Bundler.setup(:default)

require 'roda'

class App < Roda
  route do |r|
    r.root do
      'Hello World!'
    end
  end
end

run App.freeze.app

With Puma I just have to run bundle exec puma and I have a production ready service (without TLS overhead) that I can attach to my reverse proxy (h2o in my case).

What exactly would I need to do with Falcon to achieve the same? I'd much prefer to use it instead, since it scales extremely well with many connections :)

[ioquatix]

Puma is only supporting HTTP/1.1

falcon serve is almost identical in every way to puma.

If you want to use falcon host, you need to set up falcon.rb to tell falcon how to host the app, what protocol to use, what port to bind to. Then you simply run falcon host.

Maybe you can tell me what the problem is, I can help you in more detail.

[ioquatix]

I do understand more documentation is required, but you are also going to have to explain what you've tried and what's not working in order for me to help you in this specific instance.

[gottlike]

Thanks for your quick reply. I currently have 2 issues with falcon serve:

• According to the readme it's meant for development only (= less performant/stable?)
• It's HTTPS-only (with self signed cert) - how can I listen on HTTP?

[ioquatix]

@gottlike

Yes, in development mode there is a timer which causes falcon to randomly crash and/or sleep for a short duration causing a loss of performance :p

falcon serve --bind http://localhost:8080

[ioquatix]

The reason why falcon serve is for development is because it's not designed for deployment. The requirements around deployment and configuration are very different. So you should prefer falcon host where you specify the port and protocol in falcon.rb rather than on the command line.

If you are happy to specify all the arguments on the command line, there is no difference. In fact, falcon serve can be more flexible in some situations and less flexible in others (SSL, protocol selection, binding to unix pipes, etc).

[gottlike]

Yes, in development mode there is a timer which causes falcon to randomly crash and/or sleep for a short duration causing a loss of performance :p

I knew it! :joy: - But for real, there could have been inefficient extra logging or non-graceful error handling etc. :)

falcon serve --bind http://localhost:8080

Awesome, thanks! :+1:

[gottlike]

Might make sense to adapt the readme a little, so everyone just wanting a drop-in replacement for Puma knows what to do.

[ioquatix]

Yes, that's the plan.

[ioquatix]

@wtn

My interpretation is that the upstream is making a valid cleartext HTTP/2 request that's being incorrectly interpreted as HTTP/1 by falcon. I haven't investigated further to confirm/refute.

There is no way for the server to distinguish cleartext HTTP/1 from cleartext HTTP/2 in a standardised way that I'm aware of. We could try to read the first request and upgrade to HTTP/2 if it matches the HTTP/2 prefix, but that's not part of any spec that I'm aware of.

I've been working on documentation. The README was completely overburdened with information, so I've started breaking it down into guides:

https://socketry.github.io/falcon/

If you can give me feedback on it, that would be great.

I'm specifically thinking of adding a guide for users coming from Puma.

[wtn]

Thank you for explaining how to specify HTTP2 in the falcon config. I wasn't able to get the vips gem to compatible with gcc-9 or clang, but I was able to view the wiki output on Github. The current falcon behavior seems correct, so I think this issue can be closed.

The new project page and guides look amazing! I will let you know if I think of stuff to add.

[gottlike]

I'm specifically thinking of adding a guide for users coming from Puma.

:+1: - and thanks for the documentation!

[ioquatix]

Sorry, I know vips is a PITA so I removed it from the wiki, also I am no longer using the wiki but instead using a dedicated project tool: https://socketry.github.io/utopia-project/

Sorry it took this long to get it organised, but documentation is something that's been on my todo list for far too long and partly due to a lack of tooling that made sense to me.

[ioquatix]

The new project page and guides look amazing! I will let you know if I think of stuff to add.

All the source files are markdown so feel free to submit PRs too :)

[richardkmichael]

I'm doing simple experiments with different web servers regarding HTTP2 and header casing (related to Rack 3/rackup), and thought I'd try falcon.

I can't seem to get falcon to speak HTTP2 via curl. I see the wiki linked up-thread is gone, and I didn't see anything terribly specific to this situation in the falcon project-page docs.

Should any of the following result in an HTTP2 response?

(Aside, can falcon talk HTTP1.1 and HTTP2 together, on the same endpoint?)

$ cat basic.ru
run { |env| [200, {'Bad-Header' => 'oops'}, ["Hello World"]] }

$ falcon --verbose serve --bind http://localhost:9292 -c basic.ru

  0.0s     info: Falcon::Command::Serve [oid=0x7e4] [ec=0x7f8] [pid=58008] [2024-06-08 20:01:02 -0700]
               | Falcon v0.47.6 taking flight! Using Async::Container::Forked {:count=>8, :restart=>true}.
               | - Binding to: #<Falcon::Endpoint http://localhost:9292/ {}>
               | - To terminate: Ctrl-C or kill 58008
               | - To reload configuration: kill -HUP 58008
 0.01s     info: Falcon::Service::Server [oid=0x834] [ec=0x7f8] [pid=58008] [2024-06-08 20:01:02 -0700]
               | Starting server on #<Falcon::Endpoint http://localhost:9292/ {}>

Run curl, asking for HTTP2.. curl sends an upgrade header, but the connection isn't upgraded and it receives an HTTP1.1 response.

 $ curl -v --http2 'http://localhost:9292'

* Host localhost:9292 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9292...
* Connected to localhost (::1) port 9292
> GET / HTTP/1.1
> Host: localhost:9292
> User-Agent: curl/8.8.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA
>
* Request completely sent off
< HTTP/1.1 200 OK
< bad-header: oops
< vary: accept-encoding
< content-length: 11
<
* Connection #0 to host localhost left intact
Hello World 

Since I'm unsure if/how upgrade works, I thought I'd forgo it and directly talk HTTP2, but falcon doesn't like that. I notice the up-thread comment there is no specification-compliant way to detect direct HTTP2, so maybe this shouldn't be expected to work? Curl kindly blames the server. :-) (On the Falcon-side, it's the same protocol error as up-thread.)

 $ curl -v --http2-prior-knowledge 'http://localhost:9292'
* Host localhost:9292 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9292...
* Connected to localhost (::1) port 9292
* [HTTP/2] [1] OPENED stream for http://localhost:9292/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: http]
* [HTTP/2] [1] [:authority: localhost:9292]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.8.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: localhost:9292
> User-Agent: curl/8.8.0
> Accept: */*
>
* Request completely sent off
* Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Recv failure: Connection reset by peer
* Failed receiving HTTP2 data: 56(Failure when receiving data from the peer)
* Closing connection
curl: (56) Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.

# Falcon

    5m     info: Async::HTTP::Protocol::HTTP1::Request: Reading HTTP/1.1 requests for Async::HTTP::Protocol::HTTP1::Server. [oid=0x910] [ec=0x924] [pid=58026] [2024-06-08 20:06:40 -0700]
               | Headers: {} from #<Addrinfo: [::1]:51205 TCP>
    5m     info: Async::HTTP::Protocol::HTTP1::Request: PRI * from #<Addrinfo: [::1]:51205 TCP> [oid=0x910] [ec=0x924] [pid=58026] [2024-06-08 20:06:40 -0700]
               | Responding with: 200 {"bad-header"=>["oops"], "vary"=>["accept-encoding"]}; #<Protocol::Rack::Body::Enumerable length=11 body=Array> | #<Async::HTTP::Body::Statistics sent 11 bytes; took 0.73ms in total; took 0.69ms until first chunk>
    5m     warn: Async::Task: PRI * from #<Addrinfo: [::1]:51205 TCP> [oid=0x938] [ec=0x924] [pid=58026] [2024-06-08 20:06:40 -0700]
               | Task may have ended with unhandled exception.
               |   Protocol::HTTP1::InvalidRequest: "SM"
               |   → vendor/bundle/ruby/3.3.0/gems/protocol-http1-0.19.1/lib/protocol/http1/connection.rb:190 in `read_request_line'
               |     vendor/bundle/ruby/3.3.0/gems/protocol-http1-0.19.1/lib/protocol/http1/connection.rb:197 in `read_request'
               |     vendor/bundle/ruby/3.3.0/gems/async-http-0.66.3/lib/async/http/protocol/http1/request.rb:14 in `read'
               |     vendor/bundle/ruby/3.3.0/gems/async-http-0.66.3/lib/async/http/protocol/http1/server.rb:29 in `next_request'
               |     vendor/bundle/ruby/3.3.0/gems/async-http-0.66.3/lib/async/http/protocol/http1/server.rb:50 in `each'
               |     vendor/bundle/ruby/3.3.0/gems/async-http-0.66.3/lib/async/http/server.rb:50 in `accept'
               |     vendor/bundle/ruby/3.3.0/gems/io-endpoint-0.10.3/lib/io/endpoint/wrapper.rb:178 in `block in accept'
               |     vendor/bundle/ruby/3.3.0/gems/async-2.12.0/lib/async/task.rb:164 in `block in run'
               |     vendor/bundle/ruby/3.3.0/gems/async-2.12.0/lib/async/task.rb:377 in `block in schedule'

Finally, since I don't know the details of Upgrade:, I wondered if multiple requests might convince curl and falcon to switch to HTTP2 (i.e. perhaps the server is permitted to reply with HTTP1.1 and then upgrade on subsequent requests, but since there are none, curl closes the connection and doesn't bother). This didn't work either, as in the first case, I received two HTTP/1.1 responses. I guess curl explains can not multiplex, even if we wanted to because the connection wasn't upgraded to HTTP2.

$ curl -v --http2 'http://localhost:9292' 'http://localhost:9292'
* Host localhost:9292 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9292...
* Connected to localhost (::1) port 9292
> GET / HTTP/1.1
> Host: localhost:9292
> User-Agent: curl/8.8.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA
>
* Request completely sent off
< HTTP/1.1 200 OK
< bad-header: oops
< vary: accept-encoding
< content-length: 11
<
* Connection #0 to host localhost left intact
Hello World* Found bundle for host: 0x600003130270 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host localhost
> GET / HTTP/1.1
> Host: localhost:9292
> User-Agent: curl/8.8.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA
>
* Request completely sent off
< HTTP/1.1 200 OK
< bad-header: oops
< vary: accept-encoding
< content-length: 11
<
* Connection #0 to host localhost left intact