[Feature request] HTTP3 custom port and caddyfile option

[jotterbot]

Hello, it's been some time since my forum posts below:

https://caddy.community/t/experimental-http3-behind-firewall-port-forwarding/14746/8

But very much hoping to continue this conversation and get a config server option implemented to enable a http3 custom port header.

This is for a (maybe common?) use case where caddy is behind another udp/tcp load balancer (eg. AWS ELB) and listening on an address other than the default 443.

Given this basic Caddyfile:

{
    http_port 8080
    https_port 8443
    debug
}

https://localhost:8443 {
    tls internal
    respond "hello there"
}

I currently get the following output (_note the alt-svc header values 8443 matches the https_port value_):

$ curl -I https://localhost:8443
HTTP/2 200
alt-svc: h3=":8443"; ma=2592000,h3-29=":8443"; ma=2592000
server: Caddy
content-length: 11
date: Wed, 31 Aug 2022 01:09:32 GMT

Right now, i could hard code the following (since upstream supports this):

    s.h3server.Port = 443

into this line here: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/server.go#L488

If i recompile using xcaddy, i get the desired behavour (given the same Caddyfile):

curl -I https://localhost:8443
HTTP/2 200
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
server: Caddy
content-length: 11
date: Wed, 31 Aug 2022 01:10:21 GMT

It seems to me a better option though not to hardcode this value, and instead expose config to control what port is advertised.

eg.

s.h3server.Port = <value taken from json or caddyfile option TBC>

Before attempting a PR, it would be good to understand what the approach should be and/or where the config should sit. (I noticed the protocol option is deprecated for example in the code).

Keen for any thoughts - is this potentially low hanging fruit? Or more likely to be difficult to implement?

Thanks! 😄

[mholt]

Hm, interesting -- just a quick thought, have you tried setting/overwriting the Alt-Svc header using the header handler?

header Alt-Svc `h3=":443"; ma=2592000,h3-29=":443"; ma=2592000`

Dunno if this would work but worth a quick try.

[jotterbot]

Thanks for the quick response Matt!

I need to read the spec, but wouldn't then that header persist after the connection is upgraded to http3? (Maybe that is okay though?)

Also not sure how fluid that h3-29 value is and if this is a moving target with updates to the standard, or upstream lib updates?

Is there a sed or find/replace way of subbing out known header values in caddy?

[francislavoie]

Do you actually need to change http_port and https_port though? Why exactly can't you listen on 80/443 anyways? Feels like both of those things are the wrong way to go here for your usecase.

[mholt]

@francislavoie I think I get the use case. I do something similar at home where I forward ports 80 and 443 to higher ports on my machine. The outside sees and uses 80 and 443 but Caddy uses higher ports. So whenever it returns port information to the client it needs to use the port the client sees.

@jotterbot

I need to read the spec, but wouldn't then that header persist after the connection is upgraded to http3? (Maybe that is okay though?)

Yeah, no idea -- but give it a shot and see if it works for you.

Also not sure how fluid that h3-29 value is and if this is a moving target with updates to the standard, or upstream lib updates?

That is a spec draft number and will probably go away soon, is my guess; @marten-seemann would know for sure. But I would suspect he'll eventually drop draft versions and just go with h3 as soon as more clients support it.

[jotterbot]

Do you actually need to change http_port and https_port though? Why exactly can't you listen on 80/443 anyways? Feels like both of those things are the wrong way to go here for your usecase.

In addition to what @mholt says above, changing caddy back to 80/443 as a fix includes assumptions about the environment, that might not have flexibility to be changed:

Some examples:

• running caddy on system ports <1024 prohibited (for whatever reason)
• security group / firewall policies
• port conflicts (if other services are running on those ports, or in front of caddy with loopback interface)
• home router port forwarding

and most likely others...

[francislavoie]

I'm aware of all that, I'm just trying to understand why it matters for you in this case and whether there's an alternative solution for you.

[jotterbot]

Thanks @francislavoie 😄

Technically speaking, I already have a solution which is to compile/run my own binaries with the change above hardcoded. I'm not blocked in achieving what i want to via self-compile. It just seems cleaner / more scalable (for end users) to define it in config.

(I am genuinely suprised this hasn't cropped up sooner as an area of interest/enquiry for others - I am potentially running a very non-standard environment, or the deployment of HTTP3 is still niche, or both/other).

[francislavoie]

Most users (95%+ I'd postulate) can bind to ports 80/443 so it's pretty uncommon that this matters. But it's not the first time I've heard about it.

I think we probably need to add an http3_port global option, I think this would probably cover the edgecase. I don't think it ever makes sense to configure a different alt-svc port per server in Caddy, probably sufficient to configure it at the HTTP app level.

[francislavoie]

Could you test out #4997 and see if it works for you? Add http3_port 443 in your Caddyfile global options.

[mholt]

Well, hold on... 😅 Does the header directive not work?

[francislavoie]

Messing with the header is 100% a hack. It makes way more sense to have an explicit option for this, because this is what the http3.Server supports to specifically override this.

[mholt]

@francislavoie I don't think it's that much of a hack. It's a header -- and the header directive is how to set headers in Caddy. Before we add more complexity here I want to see if that works. (I haven't had a chance to test it as I'm heading to bed atm...)

I also would be interested to make sure that @jotterbot is using a Caddy build from master, as I've made significant changes to HTTP/3 and protocol-related things very recently.

[francislavoie]

I don't think it's that much of a hack. It's a header -- and the header directive is how to set headers in Caddy.

It definitely is a hack. The HTTP/3 server assembles the header value using internal behaviour. Asking users to explicitly override that is bad. It's a bad user experience because they need to find out about this problem for themselves (because if we go this route it would probably not be clearly documented, realistically) and it "disrespects" what the underlying library is doing. It's much better to take the intended "happy path" of the underlying HTTP/3 server implementation and use its Port config option. And it means we have documentation attached to this option to explain when and why it should be used.

I also would be interested to make sure that @jotterbot is using a Caddy build from master

They must be, because they don't have experimental_http3 in their quoted Caddyfile.

[jotterbot]

I will compile and test ASAP @francislavoie.

@mholt I ran this from master branch :)

Regarding complexity though, presumably I'd have to define a custom header directive for every domain/logical block i wanted to run http3 on to correct for ths wrong header? (Or, more sensibly, define a global block to import everywhere).

Would it not make more sense to use the configuration option provided for upstream? (We basically inherit all the upstream header values, as intended by the author)

[jotterbot]

Working great for me @francislavoie ! 🎉

Given this Caddyfile:

{
    http_port 8080
    https_port 8443
    http3_port 443
    debug
}

https://localhost:8443 {
    tls internal
    respond "hello there"
}

On branch http3-port-override. Running the server:

xcaddy run --config Caddyfile --watch

Check result:

$ curl -I https://localhost:8443
HTTP/2 200
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
server: Caddy
content-length: 11
date: Wed, 31 Aug 2022 05:22:36 GMT

Change value: http3_port 443 to http3_port 1234 and save Caddyfile, then validate result:

curl -I https://localhost:8443
HTTP/2 200
alt-svc: h3=":1234"; ma=2592000,h3-29=":1234"; ma=2592000
server: Caddy
content-length: 11
date: Wed, 31 Aug 2022 05:25:51 GMT

Looks good to me! 👏

[marten-seemann]

That is a spec draft number and will probably go away soon, is my guess; @marten-seemann would know for sure. But I would suspect he'll eventually drop draft versions and just go with h3 as soon as more clients support it.

It might be a while, some nodes in the libp2p ecosystem are stuck on draft-29. For Caddy, there's really no point in supporting draft versions any more. I recommend setting the supported versions in the quic.Config to QUIC v1 and v2. Happy to review a PR here (this is likely a one-line change).

[mholt]

@jotterbot But does the header directive work?

[jotterbot]

Hi @mholt , this is working using the header directive.

From latest master branch, given this Caddyfile:

{
    http_port 8080
    https_port 8443
    # http3_port 443
    debug
}

https://localhost:8443 {
    tls internal
    header alt-svc "h3=\":4321\"; ma=2592000,h3-29=\":4321\"; ma=2592000"
    respond "hello there"
}

Running xcaddy run --config Caddyfile --watch and testing with curl:

$ curl -I https://localhost:8443
HTTP/2 200
alt-svc: h3=":4321"; ma=2592000,h3-29=":4321"; ma=2592000
content-type: text/plain; charset=utf-8
server: Caddy
content-length: 11
date: Thu, 01 Sep 2022 00:11:48 GMT

I can confirm the value has changed.

[mholt]

@marten-seemann Thanks for the heads up, we may consider that -- I'll try to whip up a PR soon!

@jotterbot That's good to know, thanks so much for testing while I've been very busy these last couple days!

So here are my thoughts on things.

The Alt-Svc header is alike in almost every way to an HTTP 3xx Location header: it tells the client an address (of sorts) that it should try for service. That address is always external-facing, even if port forwarding or proxying means that the address Caddy sees/uses is different, and we have to be mindful of this.

I would prefer a zero-config approach. The good news is that we already have this with Caddy's automatic HTTP->HTTPS redirects. The Location header uses the Host header of the request to infer the port; if the port is missing it assumes 443 (reasonable, since that is the standard for HTTPS). It can use the Host header because that is the address the client knows for making the request. It doesn't require any config on our side.

What does require configuration is telling Caddy what its alternate HTTP/HTTPS ports are. Hence the http_port and https_port options. Those are internal ports, not known to clients. So it feels wrong to configure http3_port so similarly when it is an external port, especially when we can get that information from the Host header of the request automatically, without config.

And then of course we have the header directive, which sets/overrides headers. (That is its job.) I know Francis thinks it's a hack, but I disagree: it's not a hack, because that's all that the quic-go library is doing too. Just setting a header.

The header directive is so simple to use, already works, doesn't require additional config, complexity, or code changes, and IMO is the right tool for the job.

@marten-seemann How would you feel about having quic-go set the Alt-Svc header using port information from the Host header as a hint? You can see how we do it for our automatic HTTP->HTTPS redirects here:

https://github.com/caddyserver/caddy/blob/d4d8bbcfc64d1194079cae35697709f6d267d02f/modules/caddyhttp/autohttps.go#L410-L425

I think this would be more robust, as an advertised port needs to be the external address, whereas a server's listener is internal and often behind firewalls/routers that do port forwarding. Then it should "just work" for pretty much everyone, and if someone does need to override it, they can do so easily with the header directive.

(Summary: I think the best solution for this one is not to make a code change in Caddy; we have an opportunity to make the quic-go library more robust and not add a redundant config knob.)

[mholt]

@marten-seemann Btw, I just pushed a commit wherein I disable the draft version, feel free to take a look! https://github.com/caddyserver/caddy/commit/cb849bd6648294feb42eac1081aece589f20eaf6 (I tested and it does work on my machine, advertising only h3)

[JeDaYoshi]

@mholt My worry about disabling draft versions is compatibility with some clients that are not up-to-date; even if the non-draft HTTP/3 RFC was released months ago, I find this action too fast - was this taken into consideration?

[marten-seemann]

@mholt My worry about disabling draft versions is compatibility with some clients that are not up-to-date; even if the non-draft HTTP/3 RFC was released months ago, I find this action too fast - was this taken into consideration?

Horribly outdated browsers (and those are the only ones still supporting draft-29) can always fall back to TCP. There's really no need to go the extra mile to deliver extra performance to users who clearly don't care about performance (otherwise they would've updated their browser).

[marten-seemann]

@marten-seemann How would you feel about having quic-go set the Alt-Svc header using port information from the Host header as a hint? You can see how we do it for our automatic HTTP->HTTPS redirects here:

Not sure I understand how this works. The Host header is set by the client, right? So quic-go would parse the header of an incoming (potentially HTTP 1.1 / HTTP2 request), and then do what exactly? Would you assume that TCP and UDP listeners are listening on the same port, respectively?

@marten-seemann Btw, I just pushed a commit wherein I disable the draft version, feel free to take a look! https://github.com/caddyserver/caddy/commit/cb849bd6648294feb42eac1081aece589f20eaf6 (I tested and it does work on my machine, advertising only h3)

Change lgtm.

[mholt]

@JeDaYoshi Right, I agree with Marten here -- this is a cutting edge feature, it's OK if we don't support a few-versions-old Chrome for example; they will still get HTTP/2 that's pretty fast, until they upgrade their browser or it happens automatically in a few weeks/months. Since this is the bleeding edge I really don't care to support old software.

@marten-seemann Thanks for the review!

Not sure I understand how this works. The Host header is set by the client, right?

Yep. And that's the advantage: only they know the external port.

So quic-go would parse the header of an incoming (potentially HTTP 1.1 / HTTP2 request), and then do what exactly?

It would use that as a hint for the port to display on the Alt-Svc header. In a port forwarding situation (like a home network or internal firewall or something like that), only the client knows the external port it has connected to, and it puts that in the Host header.

Would you assume that TCP and UDP listeners are listening on the same port, respectively?

I suppose so... how often are they different? I'd imagine it's reasonable to assume that they are the same port by default, but it's really up to port forwarding which I don't know how to gain any insights on automatically (hence the http_port and https_port config options).

---

In other words:

Client sends request over TCP with Host: example.com, quic-go sees there is no port in the Host header, so assume :443 and put that in Alt-Svc.

If the client sends request over TCP with Host: example.com:1234, quic-go can assume that client has to connect to external port :1234, so it puts :1234 as the port in Alt-Svc.

---

This should make it work in most cases with port forwarding automatically, no config required. We just can't assume that the socket address is the same one the client connects to, because of port forwarding.

[marten-seemann]

This should make it work in most cases with port forwarding automatically, no config required. We just can't assume that the socket address is the same one the client connects to, because of port forwarding.

UDP and TCP ports happen to coincide in most cases, but just because QUIC operators decided to use UDP port 443 for HTTP/3.

I'm really not sure what best to do here. It seems like SetQuicHeaders is asked to do something that it can't deliver in the general case: it should return all the ports that QUIC listeners are listening on, but it has no way of discovering them. At the very best, with the logic that you describe, it can guess the port of one of the listeners (this is only a guess since it assume UDP port == TCP port).

Maybe the scope of SetQuicHeaders is too ambitious to begin with. Maybe it should be the caller's responsibility to pass in the port numbers (with a fallback to the ports we're listening on if none are provided maybe?). Or maybe we should have a second function where the ports are configurable by the application, and have SetQuicHeaders perform all the magic it can come up with?

[mholt]

Yeah, this is a tricky one.

I wouldn't mind quic-go "figuring out" the ports as long as it's documented how it does so, and there is also a way to tell it which ports to advertise. Ideally it'd just be a single function: I can pass in the ports, and if I don't pass in any, let quic-go make its best guess.

[timelordx]

I like the idea of Caddy filling in the port number in Alt-svc response header based on the Host request header.

[simplerick-simplefun]

I'm in the same shoe as OP, just want to share my situation&experience: I'm using HAProxy as a front load balancing proxy based on sni for my services. For my own reasons, I'm having the original tcp traffic diverted to the internal ports of each of my services, instead of having HAProxy decrypt https into http. Since I haven't figured out a way to load balance the http3/quic udp traffic(at least outside of caddy), I'm forwarding all the 443/udp traffic to my internal caddy https port(say 8443/udp) through firewall. I then rewrite the header in Caddyfile with header Alt-Svc 443 8443 to replace the h3 port caddy sends to clients from 443 to 8443.

Now it does make sense to me to have a "http3_port" option, as it makes life much easier to me: I don't need to configure firewall and edit that Alt-Svc section of the header anymore. Plus it feels like a hack to me to use iptables -t nat PREROUTING/POSTROUTING -j DNAT/SNAT to forward the http3 udp traffic. Or that I could use socat or other proxy/tunnel tools but it also feels hacking.

[mholt]

I then rewrite the header in Caddyfile with header Alt-Svc 443 8443 to replace the h3 port caddy sends to clients from 443 to 8443.

This is a great solution IMO. :man_shrugging:

[francislavoie]

I then rewrite the header in Caddyfile with header Alt-Svc 443 8443 to replace the h3 port caddy sends to clients from 443 to 8443.

That would work most of the time but it would not work in situations where the header directive does not get a chance to run before the response is written. For example, if basicauth is enabled and invalid credentials are provided, then an error response is written before header is run. You'd need to also have this inside of handle_errors routes to cover that case as well.

[bilogic]

Any idea how to remove the alt-svc header? It seems to be confusing Chrome or my sniproxy about the various services on different subdomains.

[mohammed90]

Any idea how to remove the alt-svc header? It seems to be confusing Chrome or my sniproxy about the various services on different subdomains.

This is not the right channel for this inquiry. For questions, you can use the forum https://caddy.community. The alt-svc header indicates the availability of HTTP/3. If you don't want that, disable HTTP/3 through the protocols server option.

[elee1766]

hi, i ran into this issue today.

I actually have two problems - not only was i proxying 443 to 8443, but the proxy i am using does not support UDP over ipv6 (it does support tcpv6)

this causes an issue, because it tries to upgrade to ":443", but this fails since udp over ipv6 doesn't work.

according this site https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Alt-Svc this can be resolved with a header like h3="{$PUBLIC_IPV4}:443" , and setting my public ipv4

however i mean to say - a directive that modifies such header - it should also be able to support proxying to another host, not just port. there is also the ma= note which tells the browser for how long to cache the alt-svc for, which I think also should be configurable.

perhaps a config block like

alt_svc {
  h3 {
     alt_authority <authority>
     ma <optional>
     persist <optional>
  }
}

might be more fitting ?

the clear value may be set by Header directive downstream i think.

[mholt]

@elee1766 Just use the header directive: header Alt-Svc "..."

I'll close this issue now since I can't see a reason why that header directive will not suffice for those advanced configurations. It's simple, clear, and works.