search

potats0 / apisix-coraza

3 stars 0 forks source link

Error: Test integration of this plugin to APISIX integration error #1

Open Sn0rt opened 11 months ago

Sn0rt commented 11 months ago

try to run this plugin

  1. There is no configuration case for the current plug-in, and the test cannot be without guidance.
  2. Test error.

build test ideas

  1. Use apt to install APISIX on ubuntu.
  2. Deploy the plugin.
  3. Set httpbin.org as upstream
  4. Trigger the rule (applcation injection)
  5. the error info as below.
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:280: process_request_body(): PID: 157476  phrase: access  lua-resty-coraza: failed to invoke coraza_process_request_body. msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 lua entry thread aborted: runtime error: ...local/apisix//deps/share/lua/5.1/resty/coraza/coraza.lua:223: bad argument #1 to 'coraza_intervention' (cannot convert 'nil' to 'uint64_t')
stack traceback:
coroutine 0:
    [C]: in function 'coraza_intervention'
    ...local/apisix//deps/share/lua/5.1/resty/coraza/coraza.lua:223: in function 'intervention'
    /usr/local/apisix//deps/share/lua/5.1/resty/coraza.lua:52: in function 'do_access_filter'
    /usr/local/apisix/apisix/plugins/apisix-coraza.lua:72: in function 'phase_func'
    /usr/local/apisix/apisix/plugin.lua:1097: in function 'run_plugin'
    /usr/local/apisix/apisix/init.lua:695: in function 'http_access_phase'
    access_by_lua(nginx.conf:338):2: in main chunk, client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:306: add_response_header(): PID: 157476   phrase: header_filter   lua-resty-coraza: failed to invoke coraza_add_response_header with content-length:249. msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:306: add_response_header(): PID: 157476   phrase: header_filter   lua-resty-coraza: failed to invoke coraza_add_response_header with connection:close. msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:306: add_response_header(): PID: 157476   phrase: header_filter   lua-resty-coraza: failed to invoke coraza_add_response_header with content-type:text/html. msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:306: add_response_header(): PID: 157476   phrase: header_filter   lua-resty-coraza: failed to invoke coraza_add_response_header with Server:APISIX/3.4.1. msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 [lua] coraza.lua:293: process_response_headers(): PID: 157476  phrase: header_filter   lua-resty-coraza: failed to invoke coraza_process_response_headers with 500 HTTP/1.1, msg: bad argument #1 to '?' (cannot convert 'nil' to 'uint64_t'), client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
2023/08/07 16:29:07 [error] 157476#157476: *2388 failed to run header_filter_by_lua*: ...local/apisix//deps/share/lua/5.1/resty/coraza/coraza.lua:223: bad argument #1 to 'coraza_intervention' (cannot convert 'nil' to 'uint64_t')
stack traceback:
    [C]: in function 'coraza_intervention'
    ...local/apisix//deps/share/lua/5.1/resty/coraza/coraza.lua:223: in function 'intervention'
    /usr/local/apisix//deps/share/lua/5.1/resty/coraza.lua:114: in function 'do_header_filter'
    /usr/local/apisix/apisix/plugins/apisix-coraza.lua:78: in function 'phase_func'
    /usr/local/apisix/apisix/plugin.lua:1134: in function 'common_phase'
    /usr/local/apisix/apisix/init.lua:760: in function 'http_header_filter_phase'
    header_filter_by_lua:2: in main chunk, client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"

how to reproduce

install APISIX

a new ubuntu server 22.04

echo "deb http://openresty.org/package/debian bullseye openresty" | sudo tee /etc/apt/sources.list.d/openresty.list
wget -O - https://openresty.org/package/pubkey.gpg | sudo apt-key add -
wget -O - http://repos.apiseven.com/pubkey.gpg | sudo apt-key add -
echo "deb http://repos.apiseven.com/packages/debian bullseye main" | sudo tee /etc/apt/sources.list.d/apisix.list

install APISIX

root@dev:~# apt install -y apisix
root@dev:~# apt install etcd -y

start etcd

root@dev:~# systemctl restart apisix
root@dev:~# systemctl status apisix
● apisix.service - apisix
     Loaded: loaded (/lib/systemd/system/apisix.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-08-07 15:30:21 HKT; 27s ago
    Process: 65765 ExecStart=/usr/bin/apisix start (code=exited, status=0/SUCCESS)
   Main PID: 65791 (openresty)
      Tasks: 36 (limit: 4593)
     Memory: 55.8M
        CPU: 664ms
     CGroup: /system.slice/apisix.service
             ├─65791 "nginx: master process /usr/bin/openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf"
             ├─65792 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ">
             ├─65793 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ">
             ├─65794 "nginx: cache manager process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" >
             ├─65795 "nginx: cache loader process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ">
             └─65796 "nginx: privileged agent process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" >

Aug 07 15:30:21 dev systemd[1]: Starting apisix...
Aug 07 15:30:21 dev apisix[65765]: /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua start
Aug 07 15:30:21 dev systemd[1]: Started apisix.

build coraza share lib file

follow the cmdline

$ apt-get install autoconf git libtool m4 automake make golang
$ git clone https://github.com/potats0/coraza.git
$ cd coraza
$ ./build.sh
$ ./configure
$ make
$ sudo make install

verify the lib file has been installed

root@dev:~/coraza# file /usr/local/lib/libcoraza.so /usr/local/lib/libcoraza.a
/usr/local/lib/libcoraza.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=5141ee8321285a0425f798051cebe20baedb7daf, not stripped
/usr/local/lib/libcoraza.a:  current ar archive

install lua-resty-coraza

clone potats0/lua-resty-coraza lua lib

root@dev:~# git clone https://github.com/potats0/lua-resty-coraza.git
Cloning into 'lua-resty-coraza'...
remote: Enumerating objects: 338, done.
remote: Counting objects: 100% (338/338), done.
remote: Compressing objects: 100% (193/193), done.
remote: Total 338 (delta 171), reused 235 (delta 108), pack-reused 0
Receiving objects: 100% (338/338), 53.12 KiB | 206.00 KiB/s, done.
Resolving deltas: 100% (171/171), done.

root@dev:~# cd lua-resty-coraza/

install it to APISIX deps

save the following content to lua-resty-coraza-1.0.1-0.rockspec

package = "lua-resty-coraza"
version = "1.0.1-0"
source = {
    url = "https://github.com/potats0/lua-resty-coraza",
    tag = "v1.0.1"
}

description = {
    summary = "Lua implementation of the libcoraza for modsecurity",
    homepage = "https://github.com/potats0/lua-resty-coraza",
    license = "Apache License 2.0",
    maintainer = "potats0 <bangzhiliang@gmail.com>"
}

build = {
   type = "builtin",
   modules = {
    ["resty.coraza"] = "lib/resty/coraza.lua",
    ["resty.coraza.coraza"] = "lib/resty/coraza/coraza.lua",
    ["resty.coraza.constants"] = "lib/resty/coraza/constants.lua",
    ["resty.coraza.log"] = "lib/resty/coraza/log.lua",
    ["resty.coraza.request"] = "lib/resty/coraza/request.lua",
    ["resty.coraza.response"] = "lib/resty/coraza/response.lua",
   },
}

install the lua dependency

# apt install luarocks
# cd /root/lua-resty-coraza
root@dev:~/lua-resty-coraza# luarocks make lua-resty-coraza-1.0.1-0.rockspec --tree=/usr/local/apisix/deps/

lua-resty-coraza 1.0.1-0 is now installed in /usr/local/apisix/deps (license: Apache License 2.0)

confrim

root@dev:~/lua-resty-coraza# find  /usr/local/apisix/deps| grep coraza
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0/doc
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0/doc/README_CN.md
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0/doc/README.md
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0/lua-resty-coraza-1.0.1-0.rockspec
/usr/local/apisix/deps/lib/luarocks/rocks-5.1/lua-resty-coraza/1.0.1-0/rock_manifest
/usr/local/apisix/deps/share/lua/5.1/resty/coraza
/usr/local/apisix/deps/share/lua/5.1/resty/coraza/coraza.lua
/usr/local/apisix/deps/share/lua/5.1/resty/coraza/response.lua
/usr/local/apisix/deps/share/lua/5.1/resty/coraza/log.lua
/usr/local/apisix/deps/share/lua/5.1/resty/coraza/constants.lua
/usr/local/apisix/deps/share/lua/5.1/resty/coraza/request.lua
/usr/local/apisix/deps/share/lua/5.1/resty/coraza.lua

install apisix-coraza plugin

install the APISIX coraza plugin

root@dev:~# git clone https://github.com/potats0/apisix-coraza.git
Cloning into 'apisix-coraza'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 17 (delta 3), reused 17 (delta 3), pack-reused 0
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.

install waf rule

root@dev:~# cd apisix-coraza/
root@dev:~/apisix-coraza# git submodule init
Submodule 'waf_rules' (https://github.com/coreruleset/coreruleset.git) registered for path 'waf_rules'
root@dev:~/apisix-coraza# git submodule update
Cloning into '/root/apisix-coraza/waf_rules'...
Submodule path 'waf_rules': checked out 'f2e3541cd8c18864342d2e7a0daa4db0c93f86ad'

setting APISIX, add conf/config.yaml

apisix:
    extra_lua_path: "/root/apisix-coraza/?.lua"

enable the plugin, and confirm

root@dev:~/apisix-coraza# cat /usr/local/apisix/conf/config-default.yaml |grep coraza -b10
27337-  # <- recommend to use priority (0, 100) for your custom plugins
27403-  - example-plugin                 # priority: 0
27452-  #- gm                            # priority: -43
27503-  - aws-lambda                     # priority: -1899
27556-  - azure-functions                # priority: -1900
27609-  - openwhisk                      # priority: -1901
27662-  - openfunction                   # priority: -1902
27715-  - serverless-post-function       # priority: -2000
27768-  - ext-plugin-post-req            # priority: -3000
27821-  - ext-plugin-post-resp           # priority: -4000
27874:  - apisix-coraza

reload the APISIX

$ cat logs/error.log |grep coraza -o                                                                                          [08:35:29]
coraza
coraza
coraza
coraza
coraza
coraza
coraza
coraza
coraza

test the WAF

root@dev:~/apisix-coraza# curl http://127.0.0.1:9180/apisix/admin/routes/100 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
                                                                {
                                                                    "methods": ["GET"],
                                                                    "uri": "/anything",
                                                                                 "upstream": {
                                                                        "type": "roundrobin",
                                                                        "nodes": {
                                                                      "httpbin.org:80": 1
                                                                        }
                                                                    }
                                                                }'
{"key":"/apisix/routes/100","value":{"status":1,"methods":["GET"],"priority":0,"update_time":1691396916,"uri":"/anything","create_time":1691396916,"id":"100","upstream":{"scheme":"http","type":"roundrobin","hash_on":"vars","pass_host":"pass","nodes":{"httpbin.org:80":1}}}}

inject some code 

root@dev:~/apisix-coraza# curl "http://localhost:9080/anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22" -v
*   Trying 127.0.0.1:9080...
* Connected to localhost (127.0.0.1) port 9080 (#0)
> GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1
> Host: localhost:9080
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 461
< Connection: keep-alive
< Date: Mon, 07 Aug 2023 08:28:49 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Server: APISIX/3.4.1
<
{
  "args": {
    "code": "${SHELL} -c \"echo hi\""
  },
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "localhost",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-64d0ab3e-5013a891295ce5ed30bb95c8",
    "X-Forwarded-Host": "localhost"
  },
  "json": null,
  "method": "GET",
  "origin": "127.0.0.1, 220.246.124.183",
  "url": "http://localhost/anything?code=%24{SHELL} -c \"echo hi\""
}
* Connection #0 to host localhost left intact

update the router

curl http://127.0.0.1:9180/apisix/admin/routes/100 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{
  "plugins": {
    "apisix-coraza": {
            "rules": ["REQUEST-932-APPLICATION-ATTACK-RCE"]
    }
  },
  "methods": [
    "GET"
  ],
  "uri": "/anything",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}'
potats0 commented 11 months ago

啊，稍等我更新一下最新代码

Sn0rt commented 11 months ago

啊，稍等我更新一下最新代码

目前这个插件不能正常工作.

potats0 commented 11 months ago

我更新了openresty 插件，您看看，apisix只是一个封装

potats0 commented 11 months ago

两个都更新了，不好意思啊

Sn0rt commented 11 months ago

我看你的代码和测试发现匹配不上

local schema = {
    type = "object",
    properties = {
        mode = {
            type = "string",
            default = "block",
            enum = {"block", "monitor"},
            description = "waf running at block mode or monitor mode."
        },
    }
}

测试里面多了一个 host 字段.

potats0 commented 11 months ago

host字段？

Sn0rt commented 11 months ago 
=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            local test_cases = {
                {host = "http://127.0.0.1:8181", policy = "example/allow"},
                {host = "http://127.0.0.1:8181"},
                {host = 3233, policy = "example/allow"},
            }
            local plugin = require("apisix.plugins.apisix-coraza")

            for _, case in ipairs(test_cases) do
                local ok, err = plugin.check_schema(case)
                ngx.say(ok and "done" or err)
            end
        }
    }
--- response_body
done
property "policy" is required
property "host" validation failed: wrong type: expected string, got number
potats0 commented 11 months ago

饿，host其实没完工，这个测试一直没跑起来

Sn0rt commented 11 months ago

好的吧.

potats0 commented 11 months ago

嗯嗯，有问题可以随时联系我的，我们公司上线测了一下，目前来看没啥问题

Sn0rt commented 11 months ago

你们公司测试过这个插件能正常工作? 有文档分享么?

potats0 commented 11 months ago

嗯，是的，能正常工作的，我们公司这个，类似于一个新项目，还没来得及写文档

potats0 commented 11 months ago

嗯，是的，能正常工作的，我们公司这个，类似于一个新项目，还没来得及写文档

Sn0rt commented 11 months ago

搞一个文档? 或者更新一下这个测试案例? 到目前为止我都不知道创建 router 应该传入什么参数, 全部都是根据代码分析的.

更新了代码

2023/08/08 15:02:17 [error] 629010#629010: init_worker_by_lua error: /root/apisix-coraza/apisix/plugins/apisix-coraza.lua:55: attempt to call field 'create_waf' (a nil value)
stack traceback:
    /root/apisix-coraza/apisix/plugins/apisix-coraza.lua:55: in function 'init'
    /usr/local/apisix/apisix/plugin.lua:182: in function 'load_plugin'
    /usr/local/apisix/apisix/plugin.lua:221: in function 'load'
    /usr/local/apisix/apisix/plugin.lua:340: in function 'load'
    /usr/local/apisix/apisix/plugin.lua:747: in function 'init_worker'
    /usr/local/apisix/apisix/init.lua:150: in function 'http_init_worker'
    init_worker_by_lua:2: in main chunk
2023/08/08 15:02:17 [error] 629006#629006: init_worker_by_lua error: /usr/local/apisix/apisix/plugins/apisix-coraza.lua:59: attempt to call field 'create_waf' (a nil value)
stack traceback:
    /usr/local/apisix/apisix/plugins/apisix-coraza.lua:59: in function 'init'
    /usr/local/apisix/apisix/plugin.lua:182: in function 'load_plugin'
    /usr/local/apisix/apisix/plugin.lua:221: in function 'load'
    /usr/local/apisix/apisix/plugin.lua:340: in function 'load'
    /usr/local/apisix/apisix/plugin.lua:747: in function 'init_worker'
    /usr/local/apisix/apisix/init.lua:150: in function 'http_init_worker'
    init_worker_by_lua:2: in main chunk
2023/08/08 15:02:17 [warn] 629010#629010: *16 [lua] config_etcd.lua:168: main etcd watcher started, revision=239, context: ngx.timer
2023/08/08 15:02:17 [warn] 629006#629006: *25 [lua] config_etcd.lua:168: main etcd watcher started, revision=239, context: ngx.timer
2023/08/08 15:02:18 [error] 629007#629007: *77 lua entry thread aborted: runtime error: /usr/local/apisix/apisix/init.lua:567: attempt to index upvalue 'local_conf' (a nil value)
stack traceback:
coroutine 0:
    /usr/local/apisix/apisix/init.lua: in function 'http_access_phase'
    access_by_lua(nginx.conf:338):2: in main chunk, client: 127.0.0.1, server: _, request: "GET /anything?code=%24%7BSHELL%7D%20-c%20%22echo%20hi%22 HTTP/1.1", host: "localhost:9080"
potats0 commented 11 months ago

好的，那我加你吧， 有啥需求我弄下

Sn0rt commented 11 months ago

有什么联系方式? 有邮箱么?