search

crowdsecurity / cs-nginx-bouncer

CrowdSec bouncer for Nginx
MIT License
48 stars 8 forks source link

Debian 11 (bullseye) - NGINX no more start - module 'ffi' not found #17

Closed erdoukki closed 2 years ago

erdoukki commented 2 years ago

After trying the debian package : see : https://github.com/crowdsecurity/cs-nginx-bouncer/issues/16 I am now stucked in nginx not starting because of crowdsec-nginx-bouncer !

admin@myREVERSE:~$sudo systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

sudo journalctl -xe

...
-- Subject: A start job for unit nginx.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nginx.service has begun execution.
-- 
-- The job identifier is 895.
Sep 22 11:21:06 myREVERSE nginx[3542]: nginx: [error] init_by_lua error: /usr/lib/crowdsec/lua/lrucache.lua:4: module 'ffi' not found:
Sep 22 11:21:06 myREVERSE nginx[3542]:         no field package.preload['ffi']
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/lib/crowdsec/lua/ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file './ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/share/lua/5.1/ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/share/lua/5.1/ffi/init.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/lib/lua/5.1/ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/lib/lua/5.1/ffi/init.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/share/lua/5.1/ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/share/lua/5.1/ffi/init.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file './lib/ffi.lua'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file './ffi.so'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/lib/lua/5.1/ffi.so'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/lib/aarch64-linux-gnu/lua/5.1/ffi.so'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/lib/lua/5.1/ffi.so'
Sep 22 11:21:06 myREVERSE nginx[3542]:         no file '/usr/local/lib/lua/5.1/loadall.so'
Sep 22 11:21:06 myREVERSE nginx[3542]: stack traceback:
Sep 22 11:21:06 myREVERSE nginx[3542]:         [C]: in function 'require'
Sep 22 11:21:06 myREVERSE nginx[3542]:         /usr/lib/crowdsec/lua/lrucache.lua:4: in main chunk
Sep 22 11:21:06 myREVERSE nginx[3542]:         [C]: in function 'require'
Sep 22 11:21:06 myREVERSE nginx[3542]:         /usr/lib/crowdsec/lua/CrowdSec.lua:4: in main chunk
Sep 22 11:21:06 myREVERSE nginx[3542]:         [C]: in function 'require'
Sep 22 11:21:06 myREVERSE nginx[3542]:         init_by_lua:2: in main chunk
Sep 22 11:21:06 myREVERSE nginx[3542]: nginx: configuration file /etc/nginx/nginx.conf test failed
Sep 22 11:21:06 myREVERSE systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStartPre= process belonging to unit nginx.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Sep 22 11:21:06 myREVERSE systemd[1]: nginx.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit nginx.service has entered the 'failed' state with result 'exit-code'.
Sep 22 11:21:06 myREVERSE systemd[1]: Failed to start A high performance web server and a reverse proxy server.
-- Subject: A start job for unit nginx.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nginx.service has finished with a failure.
-- 
-- The job identifier is 895 and the job result is failed.
erdoukki commented 2 years ago

Removing the package do not clean the installation :

admin@myREVERSE:~$ sudo apt remove crowdsec-nginx-bouncer
admin@myREVERSE:~$ sudo systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Purging the package get nginx start again :

admin@myREVERSE:~$ sudo apt purge crowdsec-nginx-bouncer
admin@myREVERSE:~$ sudo systemctl restart nginx
admin@myREVERSE:~$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-09-22 11:14:13 CEST; 46s ago
     Docs: man:nginx(8)
  Process: 3381 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 3382 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 3383 (nginx)
    Tasks: 3 (limit: 1191)
   Memory: 4.6M
   CGroup: /system.slice/nginx.service
           ├─3383 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─3384 nginx: worker process
           └─3385 nginx: worker process

Sep 22 11:14:13 myREVERSE systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 22 11:14:13 myREVERSE systemd[1]: Started A high performance web server and a reverse proxy server.
erdoukki commented 2 years ago

I manually install the luajit package, which may include luaffi library, but still same error !

buixor commented 2 years ago

Hello @erdoukki !

Can you tell me a bit more how you ended in this situation ? We tried on a fresh debian 11 with apt install crowdsec-nginx-bouncer and we were not able to reproduce this issue.

erdoukki commented 2 years ago

I have upgrade my Debian 10 (buster) to Debian 11 (bullseye). It is a guest lxc ARM64 based... May be a minimal rootfs is installed from lxc hosts !?

How can I help to fix ?

erdoukki commented 2 years ago

from /var/log/nginx/error.log

2021/09/22 13:17:43 [error] 3936#3936: init_by_lua error: /usr/lib/crowdsec/lua/lrucache.lua:4: module 'ffi' not found:
    no field package.preload['ffi']
    no file '/usr/lib/crowdsec/lua/ffi.lua'
    no file './ffi.lua'
    no file '/usr/local/share/lua/5.1/ffi.lua'
    no file '/usr/local/share/lua/5.1/ffi/init.lua'
    no file '/usr/local/lib/lua/5.1/ffi.lua'
    no file '/usr/local/lib/lua/5.1/ffi/init.lua'
    no file '/usr/share/lua/5.1/ffi.lua'
    no file '/usr/share/lua/5.1/ffi/init.lua'
    no file './lib/ffi.lua'
    no file './ffi.so'
    no file '/usr/local/lib/lua/5.1/ffi.so'
    no file '/usr/lib/aarch64-linux-gnu/lua/5.1/ffi.so'
    no file '/usr/lib/lua/5.1/ffi.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
    [C]: in function 'require'
    /usr/lib/crowdsec/lua/lrucache.lua:4: in main chunk
    [C]: in function 'require'
    /usr/lib/crowdsec/lua/CrowdSec.lua:4: in main chunk
    [C]: in function 'require'
    init_by_lua:2: in main chunk
2021/09/22 13:19:21 [error] 3954#3954: init_by_lua error: init_by_lua:2: module 'CrowdSec' not found:
    no field package.preload['CrowdSec']
    no file '/usr/lib/crowdsec/lua/*.lua'
    no file './CrowdSec.so'
    no file '/usr/local/lib/lua/5.1/CrowdSec.so'
    no file '/usr/lib/aarch64-linux-gnu/lua/5.1/CrowdSec.so'
    no file '/usr/lib/lua/5.1/CrowdSec.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
    [C]: in function 'require'
    init_by_lua:2: in main chunk
erdoukki commented 2 years ago 
admin@myREVERSE:~$ sudo ldd  /usr/sbin/nginx  
    linux-vdso.so.1 (0x0000007fb057d000)
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007fb03eb000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007fb03ba000)
    libcrypt.so.1 => /lib/aarch64-linux-gnu/libcrypt.so.1 (0x0000007fb0371000)
    libpcre.so.3 => /lib/aarch64-linux-gnu/libpcre.so.3 (0x0000007fb02ff000)
    libssl.so.1.1 => /usr/lib/aarch64-linux-gnu/libssl.so.1.1 (0x0000007fb0263000)
    libcrypto.so.1.1 => /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 (0x0000007faffb4000)
    libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000007faff8a000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007fafe14000)
    /lib/ld-linux-aarch64.so.1 (0x0000007fb054d000)
admin@myREVERSE:~$ sudo ldd  /usr/sbin/nginx | grep lua
erdoukki commented 2 years ago 
admin@myREVERSE:~$ sudo nginx -t
nginx: [error] init_by_lua error: /usr/lib/crowdsec/lua/lrucache.lua:4: module 'ffi' not found:
    no field package.preload['ffi']
    no file '/usr/lib/crowdsec/lua/ffi.lua'
    no file './ffi.lua'
    no file '/usr/local/share/lua/5.1/ffi.lua'
    no file '/usr/local/share/lua/5.1/ffi/init.lua'
    no file '/usr/local/lib/lua/5.1/ffi.lua'
    no file '/usr/local/lib/lua/5.1/ffi/init.lua'
    no file '/usr/share/lua/5.1/ffi.lua'
    no file '/usr/share/lua/5.1/ffi/init.lua'
    no file './lib/ffi.lua'
    no file './ffi.so'
    no file '/usr/local/lib/lua/5.1/ffi.so'
    no file '/usr/lib/aarch64-linux-gnu/lua/5.1/ffi.so'
    no file '/usr/lib/lua/5.1/ffi.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
    [C]: in function 'require'
    /usr/lib/crowdsec/lua/lrucache.lua:4: in main chunk
    [C]: in function 'require'
    /usr/lib/crowdsec/lua/CrowdSec.lua:4: in main chunk
    [C]: in function 'require'
    init_by_lua:2: in main chunk
nginx: configuration file /etc/nginx/nginx.conf test failed
buixor commented 2 years ago

Hello,

Can you show us the lua-related packages that you have installed ? I think it's more about the upgrade process than the bouncer itself

erdoukki commented 2 years ago 
admin@myREVERSE:~$ sudo apt list lua* | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

lua-ljsyscall/stable,now 0.12-1.1 all [installed]
lua-logging/stable,now 1.3.0-1.1 all [installed]
lua-nginx-memcached/stable,now 0.14-1.1 all [installed]
lua-nginx-websocket/stable,now 0.08-1 all [installed]
lua-sec/stable,now 1.0-1 arm64 [installed]
lua-socket/stable,now 3.0~rc1+git+ac3201d-4 arm64 [installed,automatic]
lua-sql-sqlite3/stable,now 2.3.4-1.1 arm64 [installed,automatic]
lua5.1/stable,now 5.1.5-8.1+b3 arm64 [installed]
lua5.3/stable,now 5.3.3-1.1+b1 arm64 [installed]
lua5.4/stable,now 5.4.2-2 arm64 [installed]
luajit/stable,now 2.1.0~beta3+dfsg-5.3 arm64 [installed]
erdoukki commented 2 years ago

I have done an upgrade of the system, and now all looks fine !

It had upgraded cs-nginx-crowdsec package and some others...

admin@myREVERSE:~$ sudo service nginx restart
admin@myREVERSE:~$ sudo service nginx status
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-09 12:30:57 CEST; 1s ago
       Docs: man:nginx(8)
    Process: 3285 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 3286 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 3287 (nginx)
        CPU: 549ms
     CGroup: /system.slice/nginx.service
             ├─3287 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─3288 nginx: worker process
             └─3289 nginx: worker process

Oct 09 12:30:56 myREVERSE systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 09 12:30:57 myREVERSE systemd[1]: Started A high performance web server and a reverse proxy server.

You may close this issue if nothing more is usefull !

I will migrate my servers to DOCKER instead of LXC, but the LXC containers may be still available a while for tests if needed !