auto_https non-intuitive behaviour

[Danack]

Hi,

I just started using Caddy and found the behaviour around auto_https quite frustrating. It may be working as intended but opening this issue as it took me hours to figure out.

I initially setup the Caddyfile with a domain name (local.phpimagick.com) that is setup at the name server to have an address of 127.0.0.1, and saw that Caddy was generating traffic to try to setup a certificate. As I didn't want to generate that traffic, or have the error messages each time I bring a local development box up, I added auto_https off.

That apparently silently stopped the site from listening on any ports. I spent quite a while trying to figure out what was happening, and went back to the simplest possible config:

  127.0.0.1 {
      respond "Hello, world!"
  }

which worked, and then re-added the auto_https off:

{
    auto_https off
}

127.0.0.1 {
    respond "Hello, world!"
}

which didn't work.......it was only then after hours of fiddling and googling, that I realised that apparently if you set auto_https off you need to explicitly disabling https when initialising:

{
    auto_https off
}

local.phpimagick.com:80 {
    respond "Hello, world !"
}

This was really not intuitive.

I realise it might be working as intended, but this seems like a massive gotcha for people new to Caddy.

Possibly just a big error message if a server isn't listening on any ports due to auto_https being set to off, would be good as having it just silently not listening to requests was hard to debug.

[mholt]

Hmm, that doesn't sound right. Maybe there's a bug?

Caddy will serve all sites over HTTPS (if you give it a host in your site block).

For IP addresses and localhost-esque domains, it will use a self-signed cert.

For all other hosts, it assumes they are public domains and will try to get a publicly-trusted cert.

HTTPS can be disabled for a site by explicitly listening on local.phpimagick.com:80 or http://local.phpimagick.com and you shouldn't need to specify auto_https off. Because you're listening on the HTTP ports, no HTTPS should be obvious. (Except the code for all this is very complicated, unfortunately. So it's possible there's a bug.)

auto_https off is a global way of disabling cert management and redirects, as if every site were on :80 or http:// by default, kind of. You shouldn't ever need both AFAIK.

I'm not able to reproduce the behavior you're seeing. For example, the following Caddyfile serves over HTTP, not HTTPS (and does not try to get any certificates):

local.phpimagick.com:80 {
    respond "Hello, world !"
}

as does this one:

http://local.phpimagick.com: {
    respond "Hello, world !"
}

Here's my log output:

$ caddy run
2022/11/06 16:53:11.768 INFO    using adjacent Caddyfile
2022/11/06 16:53:11.769 WARN    Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies    {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2022/11/06 16:53:11.769 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2022/11/06 16:53:11.769 WARN    http    server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv0", "http_port": 80}
2022/11/06 16:53:11.769 INFO    tls.cache.maintenance   started background certificate maintenance  {"cache": "0xc0009e27e0"}
2022/11/06 16:53:11.769 INFO    tls cleaning storage unit   {"description": "FileStorage:/home/matt/.local/share/caddy"}
2022/11/06 16:53:11.769 INFO    http.log    server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2022/11/06 16:53:11.769 INFO    autosaved config (load with --resume flag)  {"file": "/home/matt/.config/caddy/autosave.json"}
2022/11/06 16:53:11.770 INFO    serving initial configuration
2022/11/06 16:53:11.770 INFO    tls finished cleaning storage units

Notice how it explicitly logs that no HTTPS is being enabled.

What version of Caddy are you using? How can we reproduce the behavior you're seeing from scratch?

If it helps, I have a bug report template you can fill out.

[mholt]

That apparently silently stopped the site from listening on any ports.

To be 100% clear, do you mean that this config:

{
    auto_https off
}

local.phpimagick.com {
    respond "Hello, world !"
}

turned off all ports/listeners?

If so, that's definitely a bug. But I just tried it and Caddy is definitely serving on port 443. Of course, with auto_https off, it doesn't have a certificate so the TLS handshake fails, but there's definitely a listening socket. Turn on debug mode and you should be able to see more.

I realised that apparently if you set auto_https off you need to explicitly disabling https when initialising

Need to for what, exactly?

[Danack]

I'll put what I'm trying into a checkout-able repo, to make sure it isn't PEBKAC.

[mholt]

Sounds good, thanks for opening an issue! Let me know what you find out.

[Danack]

to make sure it isn't PEBKAC.

Pretty sure it isn't that, but also ...I think Caddy is working as you intended it to. It's just that as a developer I found the behaviour confusing, and hard to figure out. Hence the opened issue.

Test repo here https://github.com/Danack/CaddyTest where three different server configs are setup to listen on ports 8000, 8001, 8002. But as I said, it's working as intended, so possibly not much to demonstrate.

My experience was, I initially setup Caddy with

local.phpimagick.com {
    respond "Hello, world! This is caddy_0.\n\n"
}

And trying to connect to it gives:

% curl -i local.phpimagick.com:8000
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://local.phpimagick.com/
Server: Caddy
Date: Mon, 14 Nov 2022 12:47:06 GMT
Content-Length: 0

That's not going to work as I don't want to use HTTPS locally, so I googled how to disable caddy from automatically doing HTTPS, which leads to https://caddyserver.com/docs/caddyfile/options#auto-https so I change the config to have that:

{
    auto_https off
}

local.phpimagick.com {
    respond "Hello, world! This is caddy_1\n\n"
}

And now attempting to connect to the server, there is no response.

% curl -i local.phpimagick.com:8001
curl: (52) Empty reply from server

Googling some more leads to the page https://caddyserver.com/docs/automatic-https but that doesn't clearly say "Hey, if you set auto_https to be off, you probably also want to explicitly listen on port 80".

I thought I had screwed up something simiple, so was double- and triple-checking and googling for a few hours. It was only I started doing stuff at random that I got it working.

I don't think the behaviour needs to change, but maybe either:

• adding a big note to https://caddyserver.com/docs/caddyfile/options#auto-https
• if auto_https off is set, then if servers are configured without explicit ports then show a warning message ?

This seems like just a big 'gotcha' for people. I mean I'm sure going to remember it, but it seems likely to trip other people up also.

[mholt]

@Danack Thanks for the detailed feedback! Now I clearly see what you mean, and you're right, that is pretty confusing.

Admittedly, I named it poorly. Nobody had done something like this before so I didn't know what to call it...

By "auto HTTPS" we mean managing HTTPS for you, as opposed to you having to provide certs and redirects and stuff.

I think our docs are a little misleading here in bold:

When automatic HTTPS is activated, the following occurs:

• Certificates are obtained and renewed for all domain names
• The default port (if any) is changed to the HTTPS port 443
• HTTP is redirected to HTTPS (this uses HTTP port 80)

Maybe that's more like a half-truth, now that we are talking about it.

When I have a little more time I'll dive into a deeper explanation, and propose more accurate docs.

One thing to note is that it's a little weird because Caddy at its core is JSON, which does not have this abstraction you're dealing with. And some of those docs were written before I fully grasped where the "auto HTTPS" features would be implemented: whether in the core, or in the Caddyfile. I think truthfully it ended up being in a little bit of both when it comes to ports.

Because in the Caddyfile, ports are optional. In the JSON, you always have to explicitly define the ports.

Of course there are any other number of config adapters that can be used too, that might do their own thing. But most people use the Caddyfile or JSON.