test not working - Githubissues

scrazy77 commented 2 years ago

using

git clone https://github.com/jptosso/coraza-ruleset
cd coraza-ruleset
docker build . -t crs
docker run crs -name crs

not working

Dockerfile FROM jptosso/coraza-waf:latest not contains go , bash, caddy

I've change the Dockerfile to:

FROM jptosso/coraza-waf:latest
FROM golang:1.16.7-alpine
RUN apk update && apk add --no-cache bash git caddy
RUN go install github.com/fzipi/go-ftw@latest
COPY install-crs.sh .
COPY Caddyfile .
RUN ./install-crs.sh
COPY entrypoint.sh /bin/
COPY ftw.yml .
RUN touch /tmp/audit.log

ENTRYPOINT ["/bin/entrypoint.sh"]

this success build the image , but can't run the caddy.

jptosso commented 2 years ago

Hey @scrazy77 , Coraza Docker image has been upgraded and we are using busybox now. The newest Docker image does not require us to clone CRS, it's already downloaded and ready to be included.

I will update today this repository, add a readme and fix the Dockerfile.

Thank you for your feedback.

scrazy77 commented 2 years ago

Great! I want to use Coraza in Traefik. Just start study Coraza , want to write a Traefik plugin ....

jptosso commented 2 years ago

Just yesterday I created a Non-CGO version of Coraza that is supported by Traefik, Traefik uses Yaegi to interpret GO and it does not support CGO. It means if you use Coraza with Traefik you won't have @detectXSS, @detectSQLi nor PCRE expresions, hence you won't have CRS compatibility. CRS compatibility depends on the Coreruleset team replacing PCRE REGEX with RE2 REGEX (it's under development). About @detectXSS and @detectSQLi without CGO it's under early development, see https://github.com/jptosso/libinjection-go/

You can easily integrate Coraza with Traefik by using tx.ProcessRequest to handle request but you must create a ResponseWriter "interceptor" if you want to have phase 3 and 4 (https://stackoverflow.com/questions/35528330/overriding-responsewriter-interface-to-catch-http-errors)

Btw there will be an official plugin soon, it would be awesome if you share your work.

scrazy77 commented 2 years ago

That's cool ! When will the traefik official plugin release? Or anything I can do for this ? (maybe a tester ?)

jptosso commented 2 years ago

I published my Traefik draft, it should be working: https://github.com/jptosso/coraza-traefik you are more than welcome to create pull requests or issues there.

Basically it have everything a Coraza implementation needs.

Thank you!

jptosso commented 2 years ago

I have also fixed most of the Dockerfile, I have to patch the logging engine to become compatible with FTW again. Probably tomorrow.

scrazy77 commented 2 years ago

great! dockerbuild success now!

scrazy77 commented 2 years ago

now the caddy exit with route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: cannot load waf directives invalid SecRuleEngine argument

I removed the first line: SecRuleEngine DetectionOnly

better , but with: handler modules: position 0: loading module 'waf': provision http.handlers.waf: cannot load waf directives Error parsing path scanners-user-agents.data

error parsing scanners-user-agents.data.

scrazy77 commented 2 years ago

After remove all *.data rule, in /coraza/crs/custom-crs.conf the caddy runs...

jptosso commented 2 years ago

Try removing your previous image of jptosso/coraza-waf docker rmi jptosso:coraza-waf, there is a patch for SecRuleEngine and the data files. Loggers are being replaced with a new format:

SecAduitLog serial file=/tmp/audit.log

scrazy77 commented 2 years ago

great! now the test running:

➕ run 2498 total tests in 12.676438421s ⏭ skept 7 tests ☝ ignored 10 tests 👎 2236 test(s) failed to run:

scrazy77 commented 2 years ago

SecAuditLog serial file=/tmp/audit.log not working route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: cannot load waf directives invalid logger serial

I've removed image (docker rmi jptosso/coraza-waf), It seems load the old version ? I saw when building the image jptosso/coraza-waf image not downloaded... Dockerfile: FROM jptosso/coraza-waf:latest

jptosso commented 2 years ago

In the meantime, for your test version serial is called "modsec".

SecAuditLog modsec /tmp/audit.log

I am currently updating the documentation for the next Coraza version, the new syntax will be:

SecAuditLog serial file=/tmp/audit.log

About traefik:

Bad news, Coraza is not compatible with Yaegi because of the XPATH library, I will try to fork it and patch it in order to work.

It will be available this week at coraza.io

jptosso commented 2 years ago

Logging Engines

All loggers are configured using the SecAuditLog directive, you can use as many loggers as you want. You can get or create additional loggers using plugins.

# syntax is: SecAuditLog ENGINE [keyvalue options]
SecAuditLog engine_name \
              format=json \
              some_variable=some_value \
              more_setup=with_values
# or simply one line
SecAuditLog engine_name format=json some_variable=some_value

Important: Keep in mind using many loggers will lock the current routine and slow down your webserver.

Syslog

Syslog audit logger will write a TCP or UDP syslog packet to the specified host and port. Useful for SIEMs and log collectors.

network: tcp or udp (default: tcp)
format: format from logging formats (default: cef)
host: host:port (default: 127.0.0.1:514)

Example:

SecAuditLog syslog \
    network=udp \
    format=cef \
    host=somehost:514

Concurrent

Concurrent logger creates a directory structured based on the transaction's timestamp and an audit file based on the specified format. It's useful for high concurrency implementations.

format: format from logging formats (default: json)
path: absolute path where directories are going to be created (default: /opt/coraza/var/log/audit/)
dirmode: octal unix permissions for new audit directories (default: 0600)
filemode: octal unix permissions for new audit files (default: 0600)

Example:

SecAuditLog concurrent \
    format=cef \
    path=/opt/coraza/var/log/audit/ \
    dirmode=0600 \
    filemode=0600

Serial

Serial logger will write one log entry per line.

format: format from logging formats (default: cef)
path: path to the audit log (default: /opt/coraza/var/log/coraza.log)
filemode: if the file does not exist, this octal mode will be used to create it (default: 0600)

Example:

SecAuditLog serial \
    format=cef \
    path=/opt/coraza/var/log/audit.log \
    filemode=0600

Note: This audit file can be safely rotated, avoid creating giant log files as it will slow down the log writing process.

Logging Formats

JSON

This is the core structure data structure used to build any other format, it's defined

{
  "transaction": {
    "timestamp": "02/Jan/2006:15:04:20 -0700",
    "unix_timestamp": 1629575755,
    "id": "ABCDEFGHIJKMNLAB",
    "client_ip": "127.0.0.1",
    "client_port": 54481,
    "host_ip": "127.0.0.1",
    "host_port": 80,
    "server_id": "something",
    "request": {
      "method": "POST",
      "uri": "/something.php",
      "http_version": "1.1",
      "body": "some-body=with-values",
      "headers": {
        "Content-Type": ["application/x-www-form-urlencoded"],
        "Accept": ["text/html"]
      },
      "files": [{
        "name": "filename.pdf",
        "size": 1024,
        "mime": "application/pdf"
      }]
    },
    "response": {
      "status": 200,
      "headers": {
        "Set-Cookie": ["somecookie=forever; Secure", "someothercookie=wow"],
        "Content-Type": ["text/html"]
      }
    },
    "producer": {
      "connector": "github.com/jptosso/coraza-caddy",
      "version": "1.0",
      "server": "Caddy 2",
      "rule_engine": "CORERULESET/3.3",
      "stopwatch": "1417762077443733 384389; combined=20536, p1=354, p2=2901, p3=11, p4=16692, p5=578, sr=72, sw=0, l=0, gc=0"
    }
  },
  "messages": [{
    "actionset": "Warning",
    "message": "Pattern match \"\\\\< ?script\\\\b\" at ARGS_NAMES:<script.",
    "data": {
        "file": "/etc/coraza/crs/rules.conf",
        "line": 4485,
        "id": 100521,
        "rev": "1",
        "msg": "some message",
        "data": "some logdata",
        "severity": 5,
        "ver": "OWASP_CRS\/3",
        "maturity": 10,
        "accuracy": 10,
        "tags": ["some-tag", "more-tags"]
    }
  }]
}

CEF

Fields available for CEF processing:

id: Transaction id
url: Request full url
method: Request method
clientIp: Client IP address in string format
serverIp: Server IP address in string format
files: Number of files uploaded
host: Request hostname
httpVersion: HTTP version (1.1, 1.0, 2...)
reqbodylen: Request Body Length in bytes
resbodylen: Response Body Length in bytes
requestMime: Request mime type
responseMime: Response mime type
respStatus: Response status code
rules: List of rules id separated by (,).
msgs: List of messages wrapped within (") and separated by (,).

scrazy77 commented 2 years ago

About traefik:

Bad news, Coraza is not compatible with Yaegi because of the XPATH library, I will try to fork it and patch it in order to work.

It will be available this week at coraza.io

so sad... this ? https://github.com/antchfx/xpath

jptosso commented 2 years ago

Yes, this is the error I get:

 jptosso@Porco-Rosso  ~/go/coraza-traefik   master  /Users/jptosso/go/bin/yaegi test  
/Users/jptosso/go/src/github.com/jptosso/coraza-waf/vendor/github.com/antchfx/xpath/func.go:23:19: panic
panic: reflect.Set: value of type func(interface {}, string, interface {}, interface {}) bool is not assignable to type *interp.node [recovered]
    panic: reflect.Set: value of type func(interface {}, string, interface {}, interface {}) bool is not assignable to type *interp.node

goroutine 1 [running]:
github.com/traefik/yaegi/interp.runCfg.func1(0xc0003e4000, 0xc0017ab100, 0xc000b92e38)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/run.go:184 +0x253
panic(0x1a1e180, 0xc00179a5b0)
    /usr/local/Cellar/go/1.16.5/libexec/src/runtime/panic.go:965 +0x1b9
reflect.Value.assignTo(0xc0016b2b00, 0xc0005af670, 0x193, 0x1b5f588, 0xb, 0x1af92c0, 0x0, 0xc0016b2b00, 0xc0005af638, 0xc0016b2b00)
    /usr/local/Cellar/go/1.16.5/libexec/src/reflect/value.go:2451 +0x3f7
reflect.Value.Set(0x1af92c0, 0xc000a74e68, 0x196, 0xc0016b2b00, 0xc0005af670, 0x193)
    /usr/local/Cellar/go/1.16.5/libexec/src/reflect/value.go:1564 +0xbd
github.com/traefik/yaegi/interp.arrayLit.func1(0xc0003e4000, 0xc00213a930)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/run.go:2338 +0x16e
github.com/traefik/yaegi/interp.runCfg(0xc0017ab100, 0xc0003e4000)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/run.go:190 +0x87
github.com/traefik/yaegi/interp.(*Interpreter).run(0xc0003e2000, 0xc001202500, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/run.go:121 +0x2b0
github.com/traefik/yaegi/interp.(*Interpreter).importSrc(0xc0003e2000, 0xc000f2d710, 0x24, 0xc00168be21, 0x18, 0x1, 0xc001234b10, 0x3, 0x0, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/src.go:148 +0x9d4
github.com/traefik/yaegi/interp.(*Interpreter).gta.func1(0xc001408e00, 0xc001225c28)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:226 +0x13d0
github.com/traefik/yaegi/interp.(*node).Walk(0xc001408e00, 0xc000b93c28, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:233 +0xb5
github.com/traefik/yaegi/interp.(*node).Walk(0xc001408900, 0xc000b93c28, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*node).Walk(0xc001408700, 0xc000b93c28, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*Interpreter).gta(0xc0003e2000, 0xc001408700, 0xc001189b80, 0x40, 0xc00168b7c1, 0x1b, 0x0, 0x1, 0xc001408700, 0x0, ...)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:21 +0x21f
github.com/traefik/yaegi/interp.(*Interpreter).importSrc(0xc0003e2000, 0xc000b15530, 0x24, 0xc00168b7c1, 0x1b, 0x1, 0xc0013fea10, 0xc0012263a8, 0x13986c6, 0xc000263610)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/src.go:100 +0xf57
github.com/traefik/yaegi/interp.(*Interpreter).gta.func1(0xc001238500, 0xc0012269a8)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:226 +0x13d0
github.com/traefik/yaegi/interp.(*node).Walk(0xc001238500, 0xc000b949a8, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:233 +0xb5
github.com/traefik/yaegi/interp.(*node).Walk(0xc001202c00, 0xc000b949a8, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*node).Walk(0xc001202a00, 0xc000b949a8, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*Interpreter).gta(0xc0003e2000, 0xc001202a00, 0xc00168bdc0, 0x1d, 0xc000508081, 0x1d, 0x4, 0x8, 0xc001202a00, 0x0, ...)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:21 +0x21f
github.com/traefik/yaegi/interp.(*Interpreter).importSrc(0xc0003e2000, 0xc000291dba, 0x1, 0xc000508081, 0x1d, 0xc0003e2001, 0xc0002848f0, 0xc0002a27c0, 0xd4, 0x119b333)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/src.go:100 +0xf57
github.com/traefik/yaegi/interp.(*Interpreter).gta.func1(0xc00050ad00, 0xc0001a9728)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:226 +0x13d0
github.com/traefik/yaegi/interp.(*node).Walk(0xc00050ad00, 0xc000b95728, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:233 +0xb5
github.com/traefik/yaegi/interp.(*node).Walk(0xc00050a400, 0xc000b95728, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*node).Walk(0xc00050a200, 0xc000b95728, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:237 +0x66
github.com/traefik/yaegi/interp.(*Interpreter).gta(0xc0003e2000, 0xc00050a200, 0xc000291dba, 0x1, 0xc00029196c, 0x2, 0x0, 0x1, 0xc00050a200, 0x0, ...)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/gta.go:21 +0x21f
github.com/traefik/yaegi/interp.(*Interpreter).importSrc(0xc0003e2000, 0x1cd6cd8, 0x1, 0xc00029196c, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/src.go:100 +0xf57
github.com/traefik/yaegi/interp.(*Interpreter).EvalTest(...)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/interp/interp.go:427
main.test(0xc000032060, 0x0, 0x0, 0xc00000a601, 0xc000001680)
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/cmd/yaegi/test.go:151 +0x106a
main.main()
    /Users/jptosso/go/pkg/mod/github.com/traefik/yaegi@v0.9.23/cmd/yaegi/yaegi.go:135 +0x505

It's some issue with sync.Pool{} which takes New(...) interface{}

Interfaces to be used from the pre-compiled code can not be added dynamically, as it is required to pre-compile interface wrappers.

jptosso / coraza-ruleset

test not working #1

About traefik:

Logging Engines

Syslog

Concurrent

Serial

Logging Formats

JSON

CEF

About traefik: