search

teler-sh / teler-waf

teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.
https://test.teler.sh
Apache License 2.0
354 stars 32 forks source link

refactor: use single regexp engine #90

Closed dwisiswant0 closed 1 year ago

dwisiswant0 commented 1 year ago

IMPORTANT: Please do not create a PR without creating an issue first!

(Any change needs to be discussed before proceeding. Failure to do so may result in the rejection of the pull request).

Summary

Use a single regexp engine to check BadReferrer or CommonWebAttack.

Proposed of changes

This PR fixes/implements the following bugs/features:

How has this been tested?

Proof:

Before

$ go test -bench "^BenchmarkInitialize" -cpu=4 
goos: linux
goarch: amd64
pkg: github.com/kitabisa/teler-waf
cpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
BenchmarkInitializeDefault-4                              24      45378362 ns/op    44680872 B/op     114109 allocs/op
BenchmarkInitializeCommonWebAttack-4                      24      53996485 ns/op    44681077 B/op     114110 allocs/op
BenchmarkInitializeCVE-4                                  22      46210712 ns/op    44680611 B/op     114109 allocs/op
BenchmarkInitializeBadIPAddress-4                         22      48299776 ns/op    44681080 B/op     114111 allocs/op
BenchmarkInitializeBadReferrer-4                          21      48395139 ns/op    44680879 B/op     114109 allocs/op
BenchmarkInitializeBadCrawler-4                           20      50372806 ns/op    44681730 B/op     114112 allocs/op
BenchmarkInitializeDirectoryBruteforce-4                  22      47307034 ns/op    44681161 B/op     114111 allocs/op
BenchmarkInitializeWithoutCommonWebAttack-4               20      50517878 ns/op    44681520 B/op     114111 allocs/op
BenchmarkInitializeWithoutCVE-4                           22      49588731 ns/op    44680802 B/op     114111 allocs/op
BenchmarkInitializeWithoutBadIPAddress-4                  21      48446407 ns/op    44681244 B/op     114110 allocs/op
BenchmarkInitializeWithoutBadReferrer-4                   21      49127410 ns/op    44681440 B/op     114113 allocs/op
BenchmarkInitializeWithoutBadCrawler-4                    25      47271833 ns/op    44681201 B/op     114110 allocs/op
BenchmarkInitializeWithoutDirectoryBruteforce-4           22      47920421 ns/op    44680771 B/op     114109 allocs/op
BenchmarkInitializeCustomRules-4                          24      47911679 ns/op    44684078 B/op     114143 allocs/op
PASS
ok      github.com/kitabisa/teler-waf   23.858s

After

$ go test -bench "^BenchmarkInitialize" -cpu=4 
goos: linux
goarch: amd64
pkg: github.com/kitabisa/teler-waf
cpu: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
BenchmarkInitializeDefault-4                              19      54074248 ns/op    40534658 B/op      86865 allocs/op
BenchmarkInitializeCommonWebAttack-4                      20      53427788 ns/op    40534692 B/op      86865 allocs/op
BenchmarkInitializeCVE-4                                  19      56233734 ns/op    40534302 B/op      86865 allocs/op
BenchmarkInitializeBadIPAddress-4                         21      51840411 ns/op    40534723 B/op      86866 allocs/op
BenchmarkInitializeBadReferrer-4                          21      56515909 ns/op    40533931 B/op      86864 allocs/op
BenchmarkInitializeBadCrawler-4                           21      53618267 ns/op    40534408 B/op      86863 allocs/op
BenchmarkInitializeDirectoryBruteforce-4                  20      52959422 ns/op    40534349 B/op      86864 allocs/op
BenchmarkInitializeWithoutCommonWebAttack-4               21      54061027 ns/op    40534370 B/op      86865 allocs/op
BenchmarkInitializeWithoutCVE-4                           21      53972529 ns/op    40533869 B/op      86863 allocs/op
BenchmarkInitializeWithoutBadIPAddress-4                  19      54865785 ns/op    40534563 B/op      86865 allocs/op
BenchmarkInitializeWithoutBadReferrer-4                   20      52651360 ns/op    40534131 B/op      86865 allocs/op
BenchmarkInitializeWithoutBadCrawler-4                    19      53479496 ns/op    40534360 B/op      86865 allocs/op
BenchmarkInitializeWithoutDirectoryBruteforce-4           20      50101808 ns/op    40533456 B/op      86861 allocs/op
BenchmarkInitializeCustomRules-4                          20      51196911 ns/op    40537097 B/op      86895 allocs/op
PASS
ok      github.com/kitabisa/teler-waf   22.756s

Conclusion

$ benchstat old.txt new.txt 
goos: linux
goarch: amd64
pkg: github.com/kitabisa/teler-waf
cpu: 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz
                                       │   old.txt   │               new.txt                │
                                       │   sec/op    │    sec/op     vs base                │
InitializeDefault-4                      20.60m ± 5%   18.48m ± 13%  -10.29% (p=0.009 n=10)
InitializeCommonWebAttack-4              20.70m ± 5%   18.53m ±  7%  -10.49% (p=0.000 n=10)
InitializeCVE-4                          21.97m ± 6%   18.90m ±  8%  -13.99% (p=0.001 n=10)
InitializeBadIPAddress-4                 20.61m ± 6%   20.05m ± 11%        ~ (p=0.529 n=10)
InitializeBadReferrer-4                  19.97m ± 6%   19.42m ±  9%        ~ (p=0.353 n=10)
InitializeBadCrawler-4                   21.23m ± 9%   18.98m ±  7%  -10.56% (p=0.000 n=10)
InitializeDirectoryBruteforce-4          22.06m ± 9%   18.20m ±  6%  -17.46% (p=0.000 n=10)
InitializeWithoutCommonWebAttack-4       21.34m ± 4%   19.18m ±  5%  -10.13% (p=0.000 n=10)
InitializeWithoutCVE-4                   21.93m ± 7%   19.53m ±  5%  -10.94% (p=0.000 n=10)
InitializeWithoutBadIPAddress-4          20.39m ± 8%   20.07m ± 11%        ~ (p=0.052 n=10)
InitializeWithoutBadReferrer-4           20.91m ± 5%   18.97m ±  5%   -9.31% (p=0.000 n=10)
InitializeWithoutBadCrawler-4            20.63m ± 8%   19.35m ±  9%   -6.22% (p=0.003 n=10)
InitializeWithoutDirectoryBruteforce-4   20.81m ± 5%   19.18m ±  9%   -7.82% (p=0.005 n=10)
InitializeCustomRules-4                  21.53m ± 6%   17.89m ±  7%  -16.91% (p=0.000 n=10)
geomean                                  21.04m        19.04m         -9.49%

                                       │   old.txt    │               new.txt               │
                                       │     B/op     │     B/op      vs base               │
InitializeDefault-4                      42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.001 n=10)
InitializeCommonWebAttack-4              42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeCVE-4                          42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeBadIPAddress-4                 42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeBadReferrer-4                  42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeBadCrawler-4                   42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeDirectoryBruteforce-4          42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutCommonWebAttack-4       42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutCVE-4                   42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutBadIPAddress-4          42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutBadReferrer-4           42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutBadCrawler-4            42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeWithoutDirectoryBruteforce-4   42.63Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
InitializeCustomRules-4                  42.64Mi ± 0%   38.66Mi ± 0%  -9.33% (p=0.000 n=10)
geomean                                  42.63Mi        38.66Mi       -9.33%

                                       │   old.txt    │               new.txt               │
                                       │  allocs/op   │  allocs/op   vs base                │
InitializeDefault-4                      114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.001 n=10)
InitializeCommonWebAttack-4              114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeCVE-4                          114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeBadIPAddress-4                 114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeBadReferrer-4                  114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeBadCrawler-4                   114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeDirectoryBruteforce-4          114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutCommonWebAttack-4       114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutCVE-4                   114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutBadIPAddress-4          114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutBadReferrer-4           114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutBadCrawler-4            114.23k ± 0%   86.86k ± 0%  -23.96% (p=0.000 n=10)
InitializeWithoutDirectoryBruteforce-4   114.23k ± 0%   86.87k ± 0%  -23.96% (p=0.000 n=10)
InitializeCustomRules-4                  114.27k ± 0%   86.90k ± 0%  -23.95% (p=0.000 n=10)
geomean                                   114.2k        86.87k       -23.96%

Closing issues

Fixes #83

Checklist:

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 85.71% and project coverage change: -0.68% :warning:

Comparison is base (d700f3b) 79.88% compared to head (13fc733) 79.20%. Report is 1 commits behind head on master.

:exclamation: Current head 13fc733 differs from pull request most recent head 088aa0b. Consider uploading reports for the commit 088aa0b to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #90 +/- ## ========================================== - Coverage 79.88% 79.20% -0.68% ========================================== Files 5 5 Lines 1059 1034 -25 ========================================== - Hits 846 819 -27 Misses 162 162 - Partials 51 53 +2 ``` | [Files Changed](https://app.codecov.io/gh/kitabisa/teler-waf/pull/90?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kitabisa) | Coverage Δ | | |---|---|---| | [teler.go](https://app.codecov.io/gh/kitabisa/teler-waf/pull/90?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kitabisa#diff-dGVsZXIuZ28=) | `78.17% <75.00%> (-2.03%)` | :arrow_down: | | [analyze.go](https://app.codecov.io/gh/kitabisa/teler-waf/pull/90?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kitabisa#diff-YW5hbHl6ZS5nbw==) | `74.84% <100.00%> (+0.06%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dwisiswant0 commented 1 year ago

Note Please note that this PR is implemented for initializing and analyzing common web attacks and bad referrer threats.

Furthermore, there are still remaining implementations that utilize the built-in regexp engine. See grep -nr '"regexp"'.