Open cocoahero opened 2 months ago
What you are trying to do is possible, but it's probably not achievable with Async::HTTP::Internet
which is a canonical/opinionated interface for accessing internet resources following standard conventions.
More specifically, the host
header is not part of HTTP/2+, and therefore the library has a specific field for dealing with host identification, referred to as authority
. In HTTP/1, authority
maps directly to the host
header, and in HTTP/2+ it maps to the :authority
pseudo header.
Note that this can be different from the address you connect to (in your case the reverse proxy), and even different from the hostname used for TLS negotiation (can be used to select certificates for validating the connection is secure).
According to your initial problem, something like this will work for you:
server_endpoint = ::IO::Endpoint.tcp("my.reverse.proxy.com", 443)
host_endpoint = ::Async::HTTP::Endpoint.parse("https://api.example.com/path/to/resource", server_endpoint)
client = ::Async::HTTP::Client.new(host_endpoint)
client.post(...)
This will configure the authority and TLS as if it was talking to api.example.com
.
Correct me if I am wrong, but creating a client in this way will prevent connection reuse/persistent connections since we will need a Client
instance per combination of TCP endpoint and host/authority. In reality, we want to reuse the connection to the TCP endpoint, but just change the Host
header per request.
It will depend on your TLS setup, if you are expecting SNI (which is at the connection level, usually for virtual hosts) to work, you will need the setup I suggested, and yes, you will want to cache the client instances and yes, you are correct, it will be one connection at least per SNI hostname. If you don't need SNI, and only need to change the authority, you can do this instead:
def get(client, url, headers = Protocol::HTTP::Headers.new)
client.call(
Protocol::HTTP::Request.new(nil, url.hostname, "GET", url.path, headers, body)
)
end
The 2nd argument to the Request.new
is the authority (host) to use.
If this was more commonly requested, we could consider adding a keyword argument to the method helpers, e.g. internet.get(..., authority: ...)
- however I don't know if your issue encompasses SNI or not - can you give me more details about that part of your setup?
We are essentially trying to do the equivalent of the following Go code with async-http, where we want to keep a pool of persistent connections to the reverse proxy.
The reverse proxy only has a TLS certificate for my.tld.com
but has millions of virtual hosts.
We use Cloudflare SaaS where they connect to the reverse proxy/loadbalancer with TLS/SNI set to my.tld.com
and the Host/:authority
header set to the tenant's domain.
go run main.go
starting to create conn nuc.adi.run:443
starting tcp connection tcp 10.0.0.2:443
tcp connection created tcp 10.0.0.2:443 <nil>
TLS handshake done {Version:772 HandshakeComplete:true DidResume:false CipherSuite:4865 NegotiatedProtocol:h2 NegotiatedProtocolIsMutual:true ServerName:nuc.adi.run PeerCertificates:[0x140001d8008 0x140001d8588] VerifiedChains:[[0x140001d8b08 0x140001d9088 0x140001d9608]] SignedCertificateTimestamps:[] OCSPResponse:[] TLSUnique:[] ECHAccepted:false ekm:0x104d76c80 testingOnlyDidHRR:false testingOnlyCurveID:29}
connection established {Conn:0x14000192008 Reused:false WasIdle:false IdleTime:0s}
Wrote header field :authority [code.aditya.me]
starting to create conn nuc.adi.run:443
connection established {Conn:0x14000192008 Reused:true WasIdle:true IdleTime:53.75µs}
Wrote header field :authority [code.aditya.me]
Response status: 200 OK
starting to create conn nuc.adi.run:443
connection established {Conn:0x14000192008 Reused:true WasIdle:true IdleTime:82.5µs}
Wrote header field :authority [jellyfin.aditya.me]
starting to create conn nuc.adi.run:443
connection established {Conn:0x14000192008 Reused:true WasIdle:true IdleTime:30.375µs}
Wrote header field :authority [jellyfin.aditya.me]
Response status: 200 OK
Okay, I understand, SNI is not an issue for you.
Similar request has come up in the past, so I've introduced support for keyword arguments to Async::HTTP::Internet
, you can specify authority:
in your case to set the host header.
See https://socketry.github.io/async-http/releases/index.html#async::http::internet-accepts-keyword-arguments for more details.
Thanks @ioquatix, this unblocks us for now. As an aside, do you think it would also be beneficial to add a sanity check or other defensive measure to ensure the host
header is not sent twice? It was not immediately intuitive when we received 400s on the reason why, and we were only able to figure it out because we had access to nginx logs.
Unfortunately, it appears that this change does not play well with WebMock
. They use the request.authority
when building the stub signature, which results in looking for the wrong host name. https://github.com/bblimke/webmock/blob/master/lib/webmock/http_lib_adapters/async_http_client_adapter.rb#L97-L106
Thanks, I'll take a look.
While attempting to use
async-http
(0.71.0) with against a reverse proxy where we need to specify aHost
header, the library sends the header twice: one with the URL's hostname value, and one with the value specified in the request. For most web servers, this results in a 400 Bad Request.It looks like this is coming from
protocol-http1
where thehost
header is always written, irrespective if it also exists inheaders
. https://github.com/socketry/protocol-http1/blob/main/lib/protocol/http1/connection.rb#L133-L138The reverse proxy use case is one, but it could also be for any situation where the TCP/TLS socket connection should be made to a different authority than what the HTTP Host header dictates. For example, this should also be completely valid: