search

corazawaf / coraza-caddy

OWASP Coraza middleware for Caddy. It provides Web Application Firewall capabilities
https://www.coraza.io/
Apache License 2.0
290 stars 35 forks source link

Some services return a blank 200 page with Coraza enabled #71

Open ericswpark opened 1 year ago

ericswpark commented 1 year ago

Some services that I use through the reverse proxy feature on Caddy break when I enable Coraza. One example that I've run into is with Overseerr

Reproduction steps

Expected behavior

Site loads properly and redirects to /login endpoint

Actual behavior

Site returns a blank 200 page. Developer tools say that no response was returned from the server.

No logs are generated to indicate why the request has failed.

Access logs say that a 307 response code was returned, but browsers report a 200 response code (tested with Firefox and Edge).

Configuration used

Caddyfile:

{
    order coraza_waf first
}

*.example.com {
    # Comment out this entire block to make Overseerr load properly
    coraza_waf {
        load_owasp_crs
        directives `
            Include @coraza.conf-recommended
            Include @crs-setup.conf.example
            Include @owasp_crs/*.conf
            SecRuleEngine On
        `
    }

    @overseerr host overseerr.example.com
    handle @overseerr {
        reverse_proxy overseerr:5055
    }
}

Logs

Caddy stdout:

{"level":"warn","ts":1686497529.0466342,"logger":"http.handlers.waf","msg":"failed to parse server name","tx_id":"__________","error":"failed to parse server name from authority \"overseerr.example.com\", address overseerr.example.com: missing port in address"}

Caddy access.log:

{
  "level": "info",
  "ts": 1686498117.092308,
  "logger": "http.log.access.log1",
  "msg": "handled request",
  "request": {
    "remote_ip": "[redacted]",
    "remote_port": "18232",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "overseerr.example.com",
    "uri": "/login",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
      ],
// --snip--
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "overseerr.example.com"
    }
  },
  "user_id": "",
  "duration": 0.029880206,
  "size": 0,
  "status": 307,
  "resp_headers": {
    "Date": [
      "Sun, 11 Jun 2023 15:41:57 GMT"
    ],
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "X-Powered-By": [
      "Express"
    ],
    "Location": [
      "/"
    ]
  }
}
jcchavezs commented 1 year ago

Could you include in the directives and tell us about the logs?

            SecDebugLog /dev/stdout
            SecDebugLogLevel 9
ericswpark commented 1 year ago

@jcchavezs Just tried adding those lines in the directive, but I didn't see any additional lines in either the Docker logs (stdout) or the access log.

jcchavezs commented 1 year ago

Could you also I crease the log level in caddy? Coraza uses underneath the caddy log level so although you can set coraza log level to trace it depends on caddy's log level (we need to document this). Maybe set to debug?

ericswpark commented 1 year ago

@jcchavezs the only log level I found was inside the log block, which I tried to increase from INFO (the default) to DEBUG. However there was no difference and nothing else was printed to the logs, other than what I posted above.

One interesting thing I found is that only the /login endpoint seems to be "blocked" by Coraza. I initially thought it was working again and then realized Overseerr had just remembered my session. As far as I can tell the rest of the endpoints work without any issues. Are there any rules in the default ruleset that overly scrutinize endpoints with the word "login" in it?

jcchavezs commented 1 year ago

Interesting, any clue @M4tteoP @dune73

dune73 commented 1 year ago

No clue. But you may want to test with curl to be really sure what is sent across the wire.

ericswpark commented 1 year ago

OK, I checked a bit more with curl on Windows (which seems to be a PowerShell wrapper of some kind). At first I misread the output and thought everything was working as it should. So I tested some more and got the following behavior from the browsers:

So it seems like redirection is broken with Coraza, as Overseerr fails to redirect from / to /login if not logged in, and vice versa. I'm wondering if something in Coraza (or some ruleset) is rewriting something so that browsers get confused.

In fact, this is exactly what I was misreading from the curl output from before. When requesting the root / endpoint, curl returned the following:

StatusCode        : 200
StatusDescription : OK
Content           : {}
RawContent        : HTTP/1.1 200 OK
                    Alt-Svc: h3=":443"; ma=2592000
                    Transfer-Encoding: chunked
                    Date: Mon, 12 Jun 2023 12:37:21 GMT
                    Location: /login
                    Server: Caddy
                    X-Powered-By: Express

Headers           : {[Alt-Svc, h3=":443"; ma=2592000], [Transfer-Encoding, chunked], [Date, Mon, 12 Jun 2023 12:37:21
                    GMT], [Location, /login]...}
RawContentLength  : 0

We see that the 307 response sent by Caddy has been changed to 200, and the redirect location has been written to the Location: part of RawContent. I am assuming (but am not sure) that since Caddy responds with 200 the browsers don't attempt to follow this redirect location (or maybe this is not what is meant to be followed if Caddy does return a 307? Again, not sure). What I can tell is that the Coraza plugin is modifying what Caddy has initially sent, which breaks things down once Coraza is enabled.

dune73 commented 1 year ago

Funny output for curl, but I confirm that a browser is very likely to ignore a Location header if it comes with status code 200.

Also, I do not like the Transfer-Encoding header instead of Content-Length. Not illegal per se, but unusual. Apparently depends on the application or Caddy or what not.

jcchavezs commented 1 year ago

@ericswpark could you please try to reproduce the error using the example? https://github.com/corazawaf/coraza-caddy/blob/main/example/docker-compose.yml it uses httpbin which would help you to replicate whatever response the /login endpoint returns.

ericswpark commented 1 year ago

@jcchavezs sorry, but the server that I'm running the services on doesn't have Docker Compose support.

I can try and replicate it on my local machine but that might take a while. If you guys want to test on your end, it's really easy as the Overseerr service is available as a Docker image and configuration required to reach the login/main page is pretty minimal.

jcchavezs commented 1 year ago

I will try with https://docs.overseerr.dev/getting-started/installation#docker

jcchavezs commented 1 year ago

I think this is related to https://github.com/corazawaf/coraza-caddy/issues/10 cc @jptosso

KorvinSzanto commented 1 year ago

Definitely seems like this is an issue in Coraza. If I comment out all Coraza related stuff from my caddyfile I start to see my 302's flow out properly. One bizarre thing is HEAD requests flow through and get the proper 302 while GET requests get a 200 which is what causes the curl -I confusion.

jcchavezs commented 1 year ago

@ericswpark @KorvinSzanto could you please try this branch https://github.com/corazawaf/coraza-caddy/pull/85 and check if this solves your problem?

ericswpark commented 1 year ago

@jcchavezs unfortunately I am unable to test this because I moved away from Caddy altogether (I also had some other issues unrelated to Coraza that I was unable to solve in time). Sorry :(

I will keep the issue open for @KorvinSzanto and others that are still running into the bug.

KorvinSzanto commented 1 year ago

It doesn't seem to be working but perhaps we're not running xcaddy properly. Can you should the xcaddy command you'd expect us to use to test out that branch?

M4tteoP commented 1 year ago

Directly pointing to the specific commit should work fine: xcaddy build --with github.com/corazawaf/coraza-caddy/v2@5626d6c9562a5da600568acadc740b1427f043ed

KorvinSzanto commented 1 year ago

Using that commit we still see the 302 redirects flowing out as 200s so unfortunately still not working.

rorylshanks commented 12 months ago

Hi, we have also experienced this issue, and it appears to be related to the processResponse call. I have forked this repo with response processing disabled and this issue is "fixed". You can test it with

xcaddy build --with github.com/corazawaf/coraza-caddy/v2=github.com/rorylshanks/coraza-caddy/v2@main

Of course, this is not a real "fix", but rather a stopgap until this issue is fixed

jcchavezs commented 12 months ago

Hi @KorvinSzanto and @megalan247 I just tried redirection following this local env https://github.com/corazawaf/coraza-caddy#local and calling and following the redirect locations as described in https://github.com/ahmetb/go-httpbin

In the three of them I got this response from curl:

HTTP/1.1 302 Found
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Wed, 19 Jul 2023 18:59:59 GMT
Location: [redirect location]
Server: Caddy

But of course if fails if I use the reverse_proxy functionality. The problem here is that https://github.com/corazawaf/coraza-caddy/blob/main/interceptor.go#L121 this status code is never changed, meaning the Write(statusCode int) method is never been called from the reverse_proxy handler which makes me thing there is a mutation of the response somewhere directly rather than calling the http.ResponseWriter methods. Any thoughts @mholt?

UPDATE: after further investigations I realized I was doing the reverse proxy incorrectly. Now it works as expected.

mholt commented 12 months ago

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

jcchavezs commented 12 months ago

@jcchavezs Caddy wouldn't happen to be redirecting to https:// would it?

You mean because of curl -i "localhost:8080/redirect-to?url=https://google.com"? It is just an example.

jcchavezs commented 12 months ago

I just tried the following:

{
    order coraza_waf first
}

:8082 {
    log {
        output stdout
    }

    # Comment out this entire block to make Overseerr load properly
    coraza_waf {
        load_owasp_crs
        directives `
            Include @coraza.conf-recommended
            Include @crs-setup.conf.example
            Include @owasp_crs/*.conf
            SecRuleEngine On
            SecDebugLogLevel 9
        `
    }

    @overseerr host *
    handle @overseerr {
        reverse_proxy :8083
    }
}

where :8083 is just an http server that returns a 302 and a Location and it did work correctly. Could any of you @KorvinSzanto or @megalan247 come up with a caddy file and a setup where I can reproduce the error?

jptosso commented 12 months ago

This url simulates the same scenario and it works https://tosso.io/redirect?type=publication&id=2

mholt commented 12 months ago

@jcchavezs In general, Caddy redirects HTTP to HTTPS when auto-HTTPS is engaged. This happens when a site is served with a domain name, for example. So in your reproducer config above, you use the site name of :8082 which doesn't have a domain, so auto HTTPS doesn't activate. Thus there will be no redirect to HTTPS.

I'm just guessing though, since the redirect location was redacted.

rorylshanks commented 12 months ago

An example of a broken config is

{
    order coraza_waf before reverse_proxy
    auto_https off
    admin   off
    http_port 2080
    servers :2080 {

        trusted_proxies static private_ranges

    }
}

:2080 {

log {
    output stdout
    format json
    level DEBUG
}

handle * {
    coraza_waf {
        directives `
            SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
            SecRuleEngine DetectionOnly
        `
    }

    reverse_proxy https://mail.google.com {
        header_up X-Forwarded-For {header.X-Forwarded-For}
        header_down -Server
    }
}

}

And this returns for me

curl localhost:2080 -v -H "Host: mail.google.com"
*   Trying 127.0.0.1:2080...
* Connected to localhost (127.0.0.1) port 2080 (#0)
> GET / HTTP/1.1
> Host: mail.google.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=7776000
< Content-Security-Policy: frame-ancestors 'self'
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 20 Jul 2023 12:49:33 GMT
< Expires: Thu, 20 Jul 2023 12:49:33 GMT
< Location: /mail/
< Server: Caddy
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Transfer-Encoding: chunked
< 
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="/mail/">here</A>.
</BODY>
</HTML>
* Connection #0 to host localhost left intact

Notice the 200 OK with the Location header. Of course, curling https://mail.google.com directly, returns the same HTML and Location header, but with a 301 status set, which is correct.

jptosso commented 12 months ago

I'm starting to believe that as @mholt mentioned, this is a TLS issue, I added the following to tosso.io:

tosso.io www.tosso.io {
    log {
        format console
        output file /logs/access.log
        level info
    }
    handle /experiment/redirect* {
        rewrite * /{uri}
        coraza_waf {
            directives `
                SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"
                SecRuleEngine DetectionOnly
            `
        }

        reverse_proxy https://mail.google.com {
            header_up X-Forwarded-For {header.X-Forwarded-For}
            header_down -Server
            header_up Host mail.google.com
        }
    }  
}

You can try here: https://tosso.io/experiment/redirect

Curl result:

* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 301 
...........
< content-length: 244
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/experiment/redirect/mail">here</A>.
</BODY></HTML>
* Connection #0 to host tosso.io left intact

This is my Dockerfile, updated today to latest:

FROM caddy:2.6-builder-alpine AS builder
RUN xcaddy build \
    --with github.com/corazawaf/coraza-caddy/v2@latest

FROM caddy:2.6-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy