coolaj86 commented 5 years ago

Update

Caddy v2 is VULNERABLE to Domain Fronting attacks: strict_sni_host is false by default:

https://caddyserver.com/docs/json/apps/http/servers/strict_sni_host/

Original

Re: https://serverfault.com/a/908087

For me the ideal case would be:

Give a 400 Bad Request if the Hostname is not on the certificate in the TLS session (safe for HTTP/1 and HTTP/2)
Give a 400 if there's no SNI and the Hostname is not matched by an explicit directive (including wildcards) in the Caddyfile

I'm curious how Caddy handles it currently and if there are options to get the behavior I would need.

tobya commented 5 years ago

What is the current behaviour when you test this on your caddy installation?

My1 commented 5 years ago

well on my caddy it just says that there is no matching host or whatever, similarly if you were to talk to caddy using a host that isnt listed in the config (which is really annoying when I wanna access it over a local network address, and didnt plan my localhost cert to actually have the needed IPs.

tankerkiller125 commented 5 years ago

@My1 if you want a wildcard domain you can use * as the domain name in the config to make it the default config. However, with that said if the domain is not set in Caddy then caddy will not serve anything useful period. I've tried domain fronting on my own domain just for testing reasons and found that it failed completely (no default fallback site)

mholt commented 5 years ago

Caddy 1 blocks domain fronting if TLS client auth is used (since fronting could be used to bypass auth). Caddy 2 also does this, but makes it configurable (regardless of client auth).

coolaj86 commented 1 year ago

Update: Caddy v2 is VULNERABLE to Domain Fronting attacks: strict_sni_host is false by default:

https://caddyserver.com/docs/json/apps/http/servers/strict_sni_host/

francislavoie commented 1 year ago

@coolaj86 that comment is useless without more details and evidence.

coolaj86 commented 1 year ago

Zero-Day Scenario

This enables an attacker to send a different Host header than the sni servername, thereby allowing the attacker to access a site that it was not explicitly granted access to.

For example: Where dynamic tls is enabled the attacker could send aribtrary hostnames, meaning that an API that relies on caddy as a trusted proxy, which is being reversed proxy to by caddy, can be exploited with dynamic hostnames.

For example:

curl -H "Host: evil-domain.com" https://trusted-app.com/.well-known/jwks.json

If trusted-app.com is a dynamic white-label domain for SSO that does public key lookups with the assumption that the servername and hostname have been validated as aligned by a trusted proxy, then the attacker can cause the SSO to lookup their public key rather than that of an allowed domain.

I'm not entirely sure which scenarios are vulnerable and which aren't yet. When I originally opened this issue I walked away with the understanding that this attack was being prevented. My understanding may have been wrong the whole time. I'm still not clear on the current behavior, but I am clear that if SNI checking is turned off by default, that's well-known to exploitable.

coolaj86 commented 1 year ago

Caddy 1 blocks domain fronting if TLS client auth is used (since fronting could be used to bypass auth).

Caddy 2 also does this, but makes it configurable (regardless of client auth).

This is NOT true.

Caddy 2 does NOT block domain fronting by default (according to the documentation, and what I've understood from what Matt has said in a Discord chat).

mholt commented 1 year ago

What?

"does this" ==> "blocks domain fronting if TLS client auth is used"

Caddy 2 absolutely "does this." :100:

FWIW you left several important questions unanswered in our Discord chat:

[The 'ask' endpoint] is only for telling Caddy whether it's allowed to manage a certificate for a hostname. Are you using it for more than that?

and

You claim a zero-day. Can you simply show me where this is being actively exploited?

and

So let me get this straight. Your Caddyfile says, example.com { ... } and you're saying that with a HTTP Host header of something other than example.com (modulo port), the HTTP routes for example.com are still being executed?

I'll again note that the 'ask' endpoint is not an authentication mechanism and should only be used for what it's designed for: telling Caddy whether it can manage a cert for a domain name. That's all. Make any further inferences from it at your own peril, I suppose.

Where dynamic tls is enabled the attacker could send aribtrary hostnames, meaning that an API that relies on caddy as a trusted proxy, which is being reversed proxy to by caddy, can be exploited with dynamic hostnames.

There's a lot going on in this sentence. Let's unpack:

Where dynamic tls is enabled

OK, so we're talking about "on-demand TLS".

the attacker could send aribtrary hostnames

Anyone can do this anyway, regardless of on-demand TLS; but let's continue...

meaning that an API that relies on caddy as a trusted proxy

OK, so a trust decision is being made, it sounds like a backend API is expecting Caddy to do some authentication of the client.

which is being reversed proxy to by caddy

What is the config we're talking about, exactly?

with the assumption that the servername and hostname have been validated as aligned by a trusted proxy,

Why is this alignment significant to you? TLS without mutual auth performs no authentication of the client.

can cause the SSO to lookup their public key

A public key is public.

rather than that of an allowed domain.

What is deciding what domains are "allowed" and for whom? We've already established this can't be the 'ask' endpoint, unless it's being abused -- but I think the missing piece here is, what is actually performing authentication of the client?

coolaj86 commented 1 year ago

Caddy 2 also does this, but makes it configurable (regardless of client auth).

mholt commented 1 year ago

Right, it's configurable regardless of whether client auth is used or not. If client auth is used, it is enabled automatically, but you don't have to configure client auth to configure strict SNI matching.

coolaj86 commented 1 year ago

I don't care about the wordplay. I don't care about the internal implementation of the TLS plugin being abstract and separate from the HTTP plugin.

What I care about is that there's an exploit that has affected many sites and many providers many times over the years and caddy is marketed as "secure by default" (whatever the actual words are), and it's not. A simple, well-known exploit is instead considered to be a "feature".

mholt commented 1 year ago

I don't care about the wordplay.

Not really sure what you're referring to. If you want wordplay, go to Apple's website.

I don't care about the internal implementation of the TLS plugin being abstract and separate from the HTTP plugin.

Oh. :confused: Well that's a really crucial distinction to make: TLS is not HTTP. They operate at different layers, have different purposes, are totally separate classes of protocols. I know you know all this though...

What I care about is that there's an exploit that has affected many sites and many providers many times over the years and caddy is marketed as "secure by default" (whatever the actual words are), and it's not.

Can you demonstrate even one, where a bug in Caddy is allowing attackers to bypass authentication and access something they shouldn't, as you claim? If so I would love to get this fixed, but first I need to see a proof of concept with enough information to verify it's not a misconfiguration or that it's not a misunderstanding somewhere else.

My post above still stands; there's lots of missing pieces here.

My1 commented 1 year ago

I think his problem is that caddy only triggers the protection by default if TLS Auth is used, obviously auth can also be done on an application level, so if domain fronting can be used for doing something around that, it would be a problem.

mholt commented 1 year ago

So if that's the case, I definitely refute the claims that "Caddy isn't secure by default" and have even more questions.

Clearly, it seems some configuration and infrastructure has been created to grant access to something sensitive -- which I also question, since it sounds like we're talking about public keys -- but anyway, this is not Caddy's default behavior. And so, it follows that it's also the user's responsibility to configure authentication to protect something sensitive.

For example, if you set up Caddy as a file server:

example.com

root /var/www
file_server

it's "secure by default" I guess (it serves over HTTPS by default, automatically).

But then if you change your configuration to add something like:

handle /for-bob-only/* {
    reverse_proxy 10.1.2.3:1234
}

you can't argue that Caddy is doing something insecure, or that Caddy is not secure by default. You've simply created configuration that does something insecure (only Bob should be proxied, but anyone can simply by changing the request path); one can't really expect the server to know how to authenticate Bob or that it's even supposed to do authentication in the first place.

Even after discussing at length on Discord and here, I STILL have no idea what kind of authentication is being done to verify the client's identity or what kind of authorization to verify the client's permissions.

In summary, my best picture of the current situation is as follows:

The on-demand TLS 'ask' endpoint is relevant somehow. It seems it's being misused as some sort of authentication (?) mechanism instead of simply a yes/no for whether Caddy should manage a certificate for a domain name as the docs state.
Public (not private) keys are being exposed due to an "assumption that the servername and hostname have been validated as aligned by a trusted proxy," even though the documentation says this behavior depends on configuration.
The config has been withheld so we can't verify or reproduce anything.
I don't even know what is being "exploited" in Caddy or what about Caddy is "vulnerable" here, due to lack of details in the vague complaints. It sounds like you may be frustrated at misunderstanding something, which I could understand; but I'm not really sure yet what's confusing or where something went wrong.
What results have pentests against your site turned up? Is a bug in Caddy (not its configuration) revealing secrets or allowing RCE or something?

I've seen a lot of actionable security reports in my day, and this isn't one of them.

coolaj86 commented 1 year ago

@mholt If you ask me a question on discord and then block me, I can't respond.

I didn't without the config. I stopped responding because I was busy checking code to see if it was exploitable and fix loopholes. Anyway to answer you question:

If this is a zero-day, what authentication did I just bypass here? Like what was just exploited?

The public key is retrieved based on the hostname.

The hostname is supposed to be validated before it's used to construct the key url. If the hostname is not actually validated, then the attacker can verify any token.

Since I believed that caddy could be used as a trusted proxy and that it wasn't vulnerable to domain fronting attacks, I didn't have an additional hostname verification before that function was used.

I was well aware of the possibility of domain fronting attacks, which is why I had asked that question back in 2019 and my experimentation, which was based on static config, seemed to confirm that - but I didn't try between domains in the config, which probably would failed my test.

With the dynamic tls config the hostnames become completely spoofable.

Thankfully I had other checks in place which prevented the exploit from being fully realized.

Since /.well-known in my application was handled different than /api, some of the checks were not applied to /.well-known.

Bottom Line

For these X-Forwarded-* headers, by default, the proxy will ignore their values from incoming requests, to prevent spoofing.

It's paradoxical to claim

at the TLS layer, we protect against spoofing
AND at the HTTP layer, we protect against spoofing
BUT at the boundary between the two, spoofing is intentionally enabled by default

If you were just lackadaisical in regards to security in general, I'd know what to expect, but that you're pedantic about some things and completely willfully negligent in others is frustrating.

francislavoie commented 1 year ago

@coolaj86 you have to stop speaking in hypotheticals. We need an actual Caddyfile as an example, and actual curl -v commands to show how it interacts with that config. Please stop with any and all speculation until you've shown us actual evidence. It's wasting everyone's time.

Also, your tone and approach to this conversation is extremely off-putting and aggravating. Please chill out. We're much less willing to continue looking into this if you keep making personal attacks.

coolaj86 commented 1 year ago

I wasn't speaking in hypotheticals. I gave an extremely simple curl example - which you know exactly how it works because you know what Domain Fronting is.

And I explained the exact process for my real use case.

This is a technical issue. Regardless of my tactless approach, do you want to fix it or not?

Do you want to stand by

"Having [strict_sni_host] off [by default] can be a useful privacy feature for, say, domain fronting" (discord)

Or by the numerous CVEs and security articles that come up in a basic search for "Domain Fronting"?

I'm frustrated because this isn't something that "slipped through the cracks". You actually know this, by name, and actively chose to turn off a security measure on purpose.

Caddy config

Here's the stripped down config

{
    on_demand_tls {
        ask http://localhost:3001/api/webhooks/caddy/on-demand-tls/ask
    }
}

import custom-domains.Caddyfile

:443 {
    tls {
        on_demand
    }

    reverse_proxy /api/* localhost:3001
    reverse_proxy /.well-known/* localhost:3001

    handle /* {
        root * ./public/
        file_server
    }
}

But if you don't believe that Domain Fronting should be protected against by default and only optionally disabled for the rare legitimate intentional privacy, then, well, that's your thing.

mholt commented 1 year ago

(The block on Discord was mostly to drive the conversation here. Since we were discussing it on Discord, then you brought the discussion here, I wanted just one place for the conversation. Also for my mental health.)

Again, it's hard to say without details, but it sounds like your infrastructure makes some other assumptions that are actually more the crux of the issue.

at the TLS layer, we protect against spoofing AND at the HTTP layer, we protect against spoofing BUT at the boundary between the two, spoofing is intentionally enabled by default

But actually:

TLS only protects against spoofing in the direction it does authentication. Generally it protects the clients against servers spoofing their ID, but the vast majority of deployments do not authenticate clients to servers. Does yours? (We can't tell without your config, which is still a mystery to us.)
The HTTP proxy prevents spoofing by requiring you to explicitly list trusted proxies if you want to use the X-Forwarded* headers. This is not related to authentication or authorization.
First off, language like "spoofing is intentionally enabled by default" is disingenuous, and we already described why above. These protections (preventing clients from spoofing) aren't enabled on the TLS or HTTP layers by default - you have to configure trusted proxies and TLS client auth. And while you say "boundary," that makes me think more of ALPN or something. The truth is TLS and HTTP don't mix. TLS hands off to HTTP after the handshake. Anyway, the "boundary between" TLS and HTTP is a weird place to make assumptions.

What it sounds like you desire is, "server authentication in the TLS layer should authorize the client at the HTTP layer." This doesn't make sense to me for multiple reasons (breaks abstractions, non sequitur, etc) and like Francis said, we need to stop using hypothetical and get specific if we want to make progress. If not, I'll have to lock this issue as it's wasting our time and my mental energy.

Based on the information provided, merely enabling strict_sni_host would not have solved your problem anyway.

The strict_sni_host setting doesn't provide either authn OR authz. It only ensures that HTTP handlers can rely on the client credentials from the TLS handshake for the purposes of authorization. Obviously, this requires TLS client authentication. Relying on TLS ServerName and HTTP Host to be aligned without authentication + authorization is doomed to failure: that is NOT sufficient security by itself. Even with TLS client auth, the HTTP handler(s) must still do authorization!!

This is subtle, I'll grant that, and I didn't realize that myself until about 5 years ago.

But anyway, the short of it is, we'll need answers, and you'll need authentication and authorization, before there's grounds for a bug report in its present form.

My1 commented 1 year ago

problem is with most public facing things you can throw TLS Client auth right into the trash as most people wouldnt be able to use it, let alone handle the cert correctly.

I dont really see much wrong in having this active as an additional measure, or are there practical reasons to have no strict SNI/ServerName matching by default?

coolaj86 commented 1 year ago

Does Caddy protect against Domain Fronting?

The most direct answer is: Not by default.

Only in the specific case that strict_sni_host is manually enabled or tls client auth is manually enabled and strict_sni_host not manually disabled will it enforce that TLS servername and HTTP Hostname agree.

In the case of a static config file, it Domain Fronting will be limited to hostnames that are in the config file.

For dynamic / wildcard configs: not by default.

Continued...

A Spade's a Spade

The block on Discord was mostly to drive the conversation here. Since we were discussing it on Discord, then you brought the discussion here, I wanted just one place for the conversation. Also for my mental health

Couched language (and, by my estimation, passive aggressive language) is what got us here.

The world is a confusing place already. No need for mixed messages over obvious things - or over-communicating, but I guess that's my thing, so I'll do what I do.

Is your mental health actually improved by moving the conversation here? Am I not just as abrasive and blunt here as I am in discord?

I don't see any need for a couched excuse for blocking me - you did because you wanted to. That's perfectly truthful, obvious, and non-deceptive. Fine. Maybe you unblock me. Maybe you don't.

Also, the convo was already started here before you sent the last 5 messages.

If there truly is a mental health issue, I hope your mental health improves to the point that you are comfortable with direct confrontation and direction answers to direction questions - or at least to do the things that you want to do to live a more full and happy life - but for my sake, I hope it's to the point of being direct 🙂.

Why My Use Case Doesn't Matter

If you really don't get what I'm talking about, that's fine. From your response I don't think you understood what I laid out about OIDC and SSO - though you're right that there should be multiple layers of validation (though if you can trust your proxy - which I thought I could - you don't need as many).

If you want a case study of specifics, you've got the CVEs and numerous security articles.

If you don't believe in it. If it's not a use case you care about, fine. That's your thing.

My Takeaway

Having had a few similar issues pop up over the years, and trying to build a mental model around "why does caddy do X but not do Y when they appear to be tightly related", I think the answer is something like this:

if it's in an RFC, caddy may use any aspect of the RFC, regardless of what's common or well-supported
if there's not an RFC, caddy will have no opinion and may even do the thing that is regarded as bad practice as long as it is allowable and is not solely responsible as the single point of vulnerability - except for TLS ciphers

And now that I'm done I'll mute this so that I don't have to exercise self control that I don't have and can go do things that are more important than bickering over defaults and documentation.

My1 commented 1 year ago

Is your mental health actually improved by moving the conversation here? Am I not just as abrasive and blunt here as I am in discord?

I'd say not having to handle a conversation in multiple places helps in general, also obviously here on git others can take part.

mholt commented 1 year ago

(It looks like a comment was posted before mine, but GitHub didn't notify me of it. Thanks to @francislavoie for bringing it to my attention.)

I wasn't speaking in hypotheticals. I gave an extremely simple curl example - which you know exactly how it works because you know what Domain Fronting is.

Yes I know what domain fronting is. Many of our users rely on it as an important privacy feature. That's why the curl command alone is kind of useless to me. It's not really clear what you're trying to show. I nod and am like, "Yep that's what domain fronting is." Are you trying to rely on unauthenticated TLS handshakes to authorize an HTTP route??

And I explained the exact process for my real use case.

And yet neither Francis nor I have any clue what the bug is within Caddy. Everything so far points to a misconfiguration on your end.

This is a technical issue. Regardless of my tactless approach, do you want to fix it or not?

Not really, based on this issue/DMs. I'm a bit burned out over this.

In the case of a static config file, it Domain Fronting will be limited to hostnames that are in the config file.

For dynamic / wildcard configs: not by default.

This is also a bit ambiguous, so I'll respond by clarifying that it depends. Clients can put whatever domains they want in the ServerName and Host fields. It depends on your config and infrastructure that determines how those are handled in their respective layers. I see you posted a config, which I'll look at before I finish writing this reply; but so far your infrastructure has been a very black box.

Is your mental health actually improved by moving the conversation here?

It's less to keep track of, less repeating myself... but mostly it helps me to turn off when people become unprofessionally abrasive in private ("That is so dumb. Why would you knowingly enable attacks by default? This pisses me off. Seriously."). That's a boundary crossed, so I'm turning it off.

I'm frustrated because this isn't something that "slipped through the cracks". You actually know this, by name, and actively chose to turn off a security measure on purpose.

We didn't "turn it off" -- you make it sound like if we didn't do anything with our code, it would mean that TLS ServerName and HTTP Host would magically always match.

Ah! A config!

Here's the stripped down config

Sad face. :disappointed:

See, it's hard for us to gain context and understanding when things are removed.

To be very, very clear, what we need to verify a vulnerability is:

A full config with which to expose a bug in Caddy (that is not a misconfiguration)
A curl command, ideally, to demonstrate the exploit
A clear depiction of what is expected and what actually happens

Then, I can reproduce the bug on my computer, and we can discuss its validity at that point.

Right now as I look at this config (it's in 2 code blocks; intentional?), I see:

{
    on_demand_tls {
        ask http://localhost:3001/api/webhooks/caddy/on-demand-tls/ask
    }
}

import custom-domains.Caddyfile

Ok, this enables on-demand TLS, and imports a bunch of config we have no idea about, so I still can't really give answers since we don't know what the imported config is doing.

Other than the mysterious "import", nothing really interesting or wrong here.

:443 {
    tls {
        on_demand
    }

    reverse_proxy /api/* localhost:3001
    reverse_proxy /.well-known/* localhost:3001

    handle /* {
        root * ./public/
        file_server
    }
}

Ok, so this serves TLS on port 443, proxies some requests to a backend, and serves static files for everything else.

And that's all the info we have.

So looking at this, I'll nod, and say, "Yep that's a Caddy config with on-demand TLS, a reverse proxy, and some static files. Looks good!"

If I scavenge around for your earlier curl command, I get:

curl -H "Host: evil-domain.com" https://trusted-app.com/.well-known/jwks.json

It looks like this gets proxied to localhost:3001, as expected. Looks like all requests to /.well-known/* are proxied to localhost:3001 regardless of Host header. This is working as expected.

I don't see any attempt to perform authentication or authorization, and I'm still not really clear why there is a bug in Caddy.

Everything LGTM.

My conclusion is that your application was expecting authorization that was never configured, or perhaps relied on faulty assumptions for security. I'm sorry; that must have been really frustrating to discover.

Since you've muted this thread, I guess I just wasted my time replying. I will lock this thread now to prevent any further waste of time.

caddyserver / caddy

Documentation: Does Caddy protect against Domain Fronting? #2500

Update

Original

Zero-Day Scenario

Bottom Line

Caddy config

Does Caddy protect against Domain Fronting?

Continued...

A Spade's a Spade

Why My Use Case Doesn't Matter

My Takeaway