[BUG] Nginx not restarting after update with cron message (is always ON)

mrEckendonk commented 4 years ago

Describe the bug

LOCAL Version: 4.2019.10.1872 Updated: Fri Oct 18 15:53:49 SAST 2019

REMOTE Version: 4.2019.10.1873 Updated: Sat Oct 19 11:13:55 SAST 2019

Update Available => 4.2019.10.1873

Downloading: globalblacklist.conf ...[OK]

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d Nothing to update for directory: /usr/local/sbin Setting mode: 700 => /usr/local/sbin/install-ngxblocker Setting mode: 700 => /usr/local/sbin/setup-ngxblocker Setting mode: 700 => /usr/local/sbin/update-ngxblocker Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

NGINX is not running: not reloading NGINX config Emailing report to:

To Reproduce

Cronjob every 8 hours

Expected behavior

nginx is always up and running, so message NGINX is not running: not reloading NGINX config is wrong

Server (please complete the following information):

Operating System:
[X ] CentOS

Specify Exact Version of OS:

7.7

5.3.2-1.el7.elrepo.x86_64 #1 SMP Tue Oct 1 08:18:21 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

nginx version: nginx/1.17.4 (201019-195653-centos7-kvm)

nginx -V nginx version: nginx/1.17.4 (201019-195653-centos7-kvm) built by gcc 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) built with OpenSSL 1.1.1d 10 Sep 2019 TLS SNI support enabled configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -lpcre -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=4 -fuse-ld=gold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -Wno-error=strict-aliasing -fstack-protector-strong -flto=4 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=201019-195653-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-http_flv_module --with-http_mp4_module --add-module=../nginx-rtmp-module --with-libatomic --with-http_gzip_static_module --with-file-aio --add-dynamic-module=../ngx_brotli --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-http_realip_module --add-module=../nginx-http-concat-master --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-dynamic-module=../lua-nginx-module-0.10.15 --add-module=../stream-lua-nginx-module-0.0.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.1.1d --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold'



 - Other Environments [include Version information]

 - [X ] Other
NONE (CENTMIN)

itoffshore commented 4 years ago

The helper scripts search for binaries in:

/bin /usr/bin /usr/local/bin /usr/sbin /usr/local/sbin /root/bin /root/.bin

you have nginx running somewhere else ?

mrEckendonk commented 4 years ago

Unsure it runs here when I look at /etc/init.d/nginx

nginx="/usr/local/sbin/nginx"

and /usr/local/sbin/nginx contains

ls -lha
total 52M
drwxr-xr-x.  2 root root  174 Oct 20 19:57 .
drwxr-xr-x. 21 root root  258 Oct  3 09:01 ..
-rwx------   1 root root 9.9K Oct  4 18:44 install-ngxblocker
lrwxrwxrwx   1 root root   27 Oct  3 09:01 lmd -> /usr/local/maldetect/maldet
lrwxrwxrwx   1 root root   27 Oct  3 09:01 maldet -> /usr/local/maldetect/maldet
-rwxr-xr-x   1 root root 5.0M Oct 20 19:57 nginx
-rwxr-xr-x   1 root root 5.0M Oct 20 12:56 nginx.old
-rwxr-xr-x   1 root root  11M Oct 20 10:06 php-fpm
-rwxr-xr-x   1 root root  31M Oct 20 10:06 php-fpm-b4strip
-rwx------   1 root root  14K Oct  4 18:48 setup-ngxblocker
-rwx------   1 root root  14K Oct  4 18:48 update-ngxblocker

But this is v4 install I think. On other servers (I destroyed most of them) it worked always fine in the same configuration.

itoffshore commented 4 years ago

to find the path to nginx in a shell run:

which nginx

in centos you may need to install which as it is not available by default

mrEckendonk commented 4 years ago

Install by default on Centminmod (also almost ready for centos8)

which nginx
/usr/local/sbin/nginx

itoffshore commented 4 years ago

you need to work out why this command on line 361 is failing on your system:

pidof nginx

mrEckendonk commented 4 years ago

line 361 from??

pidof nginx 15197 15196 2175

itoffshore commented 4 years ago

line 361 of the update script

I think the problem is either your cron job or it's environment

a simple test is to run the following:

#!/bin/sh
if pidof nginx 1>/dev/null; then
  echo "my cron is the problem"
fi

as it is the same check that is done by the update script

mrEckendonk commented 4 years ago

OK ./test.sh my cron is the problem

task is

00 */8 * * * /usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e <<SNIPPED>>

mrEckendonk commented 4 years ago

Fix... path in cron.... 00 */8 * * * /usr/local/sbin/nginx/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e <<SNIPPED>>

Keep issue open to see if it is resolve

I was already wondering.... Send new Google Ghost, update and he gets back.

itoffshore commented 4 years ago

I think you should just place your binaries in /usr/local/sbin & not in /usr/local/sbin/nginx/

ln -s /usr/local/sbin/nginx/nginx /usr/local/sbin/nginx

would also fix it so the update script detects your nginx in a non standard location

mrEckendonk commented 4 years ago

Thx, will create link and change/check the cron.

I could not check if it was working, we have setup a postfix mail server, but we have some issues. I don't receive any system mails also my contribution for the Google Ghost for that server has stopped.

But it keeps strange, always I use the same setup on my servers and never had this issue

mrEckendonk commented 4 years ago

Here we go......

ln -s /usr/local/sbin/nginx/nginx /usr/local/sbin/nginx ln: failed to create symbolic link ‘/usr/local/sbin/nginx’: File exists

lol

itoffshore commented 4 years ago

this is a misconfigured server not a bug - so closing this issue

mrEckendonk commented 4 years ago

this is a NOT misconfigured server, just a little different configuration. It works, updates and now even stops new bots. Test from other servers. so closing it is OK

mitchellkrogza commented 4 years ago

So its fixed now? @Eckybrazzz

mrEckendonk commented 4 years ago

@mitchellkrogza Guess so, someone installed a mail server, so have a conflict. I don't I don'1 reveive system emails. So don't know if the message is still there.

But tested it on commit, looked what has been the latest commit, tested from other a server and it gets blocked.

MeiRos commented 4 years ago

Hi @Eckybrazzz , @mitchellkrogza and @itoffshore !

I have the same problem like Eckybrazzz. But I think the troublemaker is pidof command. Because the email send to my gmail is a little bit different than the email in local box and read with Mutt.

This is gmail.

LOCAL Version: 4.2020.01.1936
Updated: Thu Jan 23 13:08:54 SAST 2020

REMOTE Version: 4.2020.01.1937
Updated: Fri Jan 24 21:32:54 SAST 2020

Update Available => 4.2020.01.1937

Downloading: globalblacklist.conf ...[OK]

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

NGINX is not running: not reloading NGINX config
Emailing report to:my@email.com

This is mutt

LOCAL Version: 4.2020.01.1936
Updated: Thu Jan 23 13:08:54 SAST 2020

REMOTE Version: 4.2020.01.1937
Updated: Fri Jan 24 21:32:54 SAST 2020

Update Available => 4.2020.01.1937

Downloading: globalblacklist.conf ...[OK]

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
/usr/local/sbin/update-ngxblocker: line 361: pidof: command not found

NGINX is not running: not reloading NGINX config
Emailing report to:my@email.com

Mutt shows error. /usr/local/sbin/update-ngxblocker: line 361: pidof: command not found Gmail don't show it.

I edited /usr/local/sbin/update-ngxblocker. The line 361 looks now like this. if /usr/sbin/pidof nginx 1>/dev/null; then After adding path to pidof the email looks like this.

LOCAL Version: 4.2020.01.1937
Updated: Fri Jan 24 21:32:54 SAST 2020

REMOTE Version: 4.2020.01.1938
Updated: Sat Jan 25 21:33:45 SAST 2020

Update Available => 4.2020.01.1938

Downloading: globalblacklist.conf ...[OK]

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
Nothing to update for directory: /usr/local/sbin
Setting mode: 700 => /usr/local/sbin/install-ngxblocker
Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
Setting mode: 700 => /usr/local/sbin/update-ngxblocker
Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

Reloading NGINX configuration...[OK]
Emailing report to: my@email.com

And it looks the same with mutt and gmail.

I'm also using Centminmod. My cron runs the update. 11 1 * * * /usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e my@email.com Bad bot blocker have been in use about 2 weeks and update only updates globalblacklist.conf but I think that's the way it should be going.

itoffshore commented 4 years ago

this commit should solve this issue once & for all

MeiRos commented 4 years ago

Sorry to bother you, but what is the recommended way to update other files than globalblacklist.conf?

mitchellkrogza commented 4 years ago

Should be no need to update any of the other includes as they never (seldom) ever change. Also updating them would overwrite any customizations you have done. globalblacklist is the one and only central heart beat that should be kept up to date.

mitchellkrogza / nginx-ultimate-bad-bot-blocker