FR: Flag health errors for self-signed or otherwise invalid certificates

[DentonGentry]

Tell us about your idea!

Provide a health signal when connectivity is blocked by a TLS MITM self-signed certificate

What are you trying to do?

We have had a few instances where connectivity has been interrupted due to a security appliance deciding to MITM tailscaled's connections to Control or DERP servers. The most recent incident was when a Fortigate config changed, and TLS connections to derp4 and log.tailscale.io failed because the certificate was self-signed.

A self-signed certificate is never right when speaking to production, we should flag that as a health check error.

How should we solve this?

No response

What is the impact of not solving this?

No response

Anything else?

No response

Front conversations

[bradfitz]

Or: we [optionally] allow it and just re-TLS handshake over the untrusted TLS channel.

[DentonGentry]

TLS-for-keepsies inside TLS-for-fakesies?

If that works it would be brilliant, but it sounds like a 1990s hacker movie.

[danderson]

Such a cert is likely to also lead to the control connection failing, so it's unclear to me how useful this'll be. We can certainly log it, but a health status seems likely to not propagate for the very reason we're logging about.

[DentonGentry]

If we can build it in a way to signal to the user that they're offline because they are experiencing some kind of MITM, that would also be good.

In the case which prompted filing this, we were not getting control or logs connections. We got onto a videoconference to look at local logs, and they were full of complaints about a self-signed certificate that had "Fortigate" in its name.

[mihaip]

My local library appears to have introduced ContentKeeper on its WiFi, which is attempting to do some kind of SSL MITM. Even with #7214 (running 1.37.135) I'm not able to have Tailscale successfully establish a connection with control, it fails with a x509: certificate signed by unknown authority error. Bug report at BUG-2170128b2685f0d55872528a2732875013d69c81f7eaaeb98554137d4adf722e-20230218221708Z-c941b8e967746073

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
# Health check:
#     - not logged in, last login error=fetch control key: Get "https://controlplane.tailscale.com/key?v=57": x509: certificate signed by unknown authority
#     - This is an unstable (development) version of Tailscale; frequent updates and bugs are likely

unexpected state: NoState

This is what Chrome sees as the cert for controlplane.tailscale.com:

[andrew-d]

Looks like #7149 works correctly in this case:

tlsdial: error: server cert for "controlplane.tailscale.com" failed to verify and is not a Let's Encrypt cert
tlsdial: error: server cert for "controlplane.tailscale.com" failed to verify and is not a Let's Encrypt cert
tlsdial: error: server cert for "derp2d.tailscale.com" failed to verify and is not a Let's Encrypt cert
tlsdial: error: server cert for "controlplane.tailscale.com" failed to verify and is not a Let's Encrypt cert

It's not in the health output since we exit early from overallErrorLocked when not logged in.

[andrew-d]

And this is (unfortunately) expected even with #7214; the initial request to control to fetch the Noise key needs to go over verified TLS, even if using that key doesn't. I wonder if we should cache the most recent server key on-disk so #7214 would allow communication if you've ever successfully started Tailscale previously (versioned by the current capver)?

[cliss]

I'm sitting at my local library as we speak, and I am having this exact same issue. Are there any reliable workarounds?