Caddy fails to install root certificate on linux when using caddy.service

[tim-hm]

Hi firstly longtime nginx user who has just discovered caddy ... loving it so thank you 🙏

Problem

Caddy is unable to install the root cert on Ubuntu 20.10 installation when following the install process. As you can see from the logs below, the issue is that the provided caddy user does not have permissions to write to the certificate stores:

Jul 19 20:59:32 ubuntu-vm sudo[10043]: pam_unix(sudo:auth): conversation failed
Jul 19 20:59:32 ubuntu-vm sudo[10043]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Jul 19 20:59:32 ubuntu-vm sudo[10043]:    caddy : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_158313951851744774645843564988098209363.crt

Reproduce

• Install fresh ubuntu 20.10 in a vm
• Install caddy and other prereqs
• Check that the caddy service has started with systemctl status caddy and if not start it
• Run curl https://localhost and you should get a certificate error

# Caddyfile
localhost {
  respond "hi"
}

What I've tried

If I run sudo caddy run then certificates work correctly for curl but not Firefox or chrome. However, this doesn't resolve the issue for the caddy user because the cert is in the root user's nssdb and I'd like to get it working from the browser for local dev. If I change the user inside /lib/systemd/system/caddy.service to root then the cert is installed correctly for curl but not for firefox or chrome (haven't tested JAVA yet). Running caddy trust as root and then starting the service does work either.

Workaround

So for the moment I've changed the caddy.server user to root, which successfully installs the cert when it starts up and then i use certutil to manually install the certs for firefox and chrome and keytool to import it for java.

[mholt]

As the message says, you could add the caddy user to your sudoers. Or run caddy trust with your own account permissions.

[tim-hm]

Afaik systemctl won't prompt for a password for the user the unit is running ... so you'd need to set caddy to run privileged commands without a password via the sudoers file. I'd like to avoid that. These are the relevant logs when you add caddy to the sudo group:

Jul 19 22:44:38 ubuntu-vm sudo[3984]: pam_unix(sudo:auth): Couldn't open /etc/securetty: No such file or directory
Jul 19 22:44:38 ubuntu-vm sudo[3984]: pam_unix(sudo:auth): conversation failed
Jul 19 22:44:38 ubuntu-vm sudo[3984]: pam_unix(sudo:auth): auth could not identify password for [caddy]

Perhaps it might be worth updating the documentation to highlight the issue for linux users?

However even when running caddy.server as root and running caddy trust as my user, firefox, chrome and java still require manual installation. Is that expected?

[francislavoie]

It only needs to be done once, so you can run it like this:

sudo HOME=~caddy caddy trust

This sets the HOME env var to the caddy user's HOME, then runs caddy trust with sudo permissions which is necessary to write the root CA cert to the system trust store.

And yes, I agree this should be documented here https://caddyserver.com/docs/command-line#caddy-trust

[mholt]

I don't really want our documentation to start getting out of scope. That is about how to use the computer more than it is about how to use Caddy.

[francislavoie]

Well, we have conflicting recommendations. If we recommend people use our systemd service to run Caddy, then it makes the default behaviour of installing the root CA cert in the system's trust store not work. So it should be documented as a workaround.

The workaround is also very much not obvious unless you read the code and notice that the relevant piece of information the caddy trust command needs is HOME to be set to that of the user under which Caddy will actually be run. So it's definitely not just a "how to use your computer" situation, it's a domain-specific workaround which is only relevant to this particular situation in Caddy.

[tim-hm]

Not sure this is an issue of "how to use a computer" more so an issue around the documentation being misleading. Other than launching caddy, the docs emphasizes that setup occurs with minimal/no user interaction, but having to edit original unit files and run certutils yourself is kind of the opposite experience.

When the main Linux distributions are listed as "Official" in the install section, its reasonable for the documentation to highlight the issue.

[mholt]

Ensuring your system is configured with proper permissions is just as much not Caddy-specific as opening ports 80 and 443 on your firewall to serve HTTP/HTTPS. It's common sense when you know how your computer works. We document what Caddy does, but if we start documenting how external systems work, we'd escape scope real fast.

But it could definitely be in our wiki.

[MestreLion]

Considering how Caddy "uses the computer" when running as a systemd unit, a unit provided by caddy in caddy's .deb from caddy's official repositories, the infrasructure / paths created and required by caddy should be part of caddy official documentation.

All of its Tutorials are using the CLI, which is great for learning purposes. But when running in production using a proper systemd file, it's a whole new game. It creates a whole environment, such as custom caddy user/group, $HOME at /var/lib/caddy, config at /etc/caddy/Caddyfile, etc, etc.

So all those paths and behavior and requirements for the especific case of using the official distribution package should be properly documented, not merely hinted at in scattered places.

And, please, add a man page for caddy!

[mholt]

I agree we could probably better document what the systemd unit file does in terms of HOME and the config path.

As for a man page, how do you recommend we do that with the dynamic CLI/plugins? (Probably best discussed in a new issue, however.)

[francislavoie]

man page is being worked on https://github.com/caddyserver/dist/pull/73

[francislavoie]

@MestreLion I've made a commit to the docs to update the https://caddyserver.com/docs/running#linux-service page to emphasize the storage locations. See https://github.com/caddyserver/website/commit/79521041e80221a5f36f0aa76de8bd2d68bdc56c. It should be live soon (as soon as @mholt has a change to push the changes to the website)

Back on the topic of the caddy trust command, we have some ideas to make this easier, and you can follow https://github.com/caddyserver/caddy/issues/4058#issuecomment-917429284 for updates about that. I'll need to work with @mholt when he has the time to implement the changes for this.

[mholt]

Deployed that page to the site

[MestreLion]

You guys are incredibly responsive, thanks! (and congrats!)

I didn't imagine my grumbling on a (closed) issue, mentioning a somewhat off-topic problem, would generate this amount of responses and commits. I'll take a look on all issues, commits and docs you guys pointed out, and try to give feedback (on the appropriate threads this time)

Once again, thank you so much for this attention!

(PS: Caddy is amazing. Had some trouble adjusting from my Apache/Nginx "mindset", and still have some nitpicks, but man, what a sweet software. Loving the Caddyfile format as well as the automatic HTTP certificate issuing. Didin't use the API yet but looking forward to. So far, so great!)

[mholt]

Thanks for the great feedback, @MestreLion -- we try our best. :)