IMPORTANT UPDATE - include files !!!!

[mitchellkrogza]

Please note the following include files have been updated today to include the required word boundary regex formatting.

bad-referrer-words.conf
blacklist-domains.conf
custom-bad-referrers.conf
whitelist-domains.conf

Please make sure to update your include files but PLEASE backup your existing files so that you do not lose any customizations you have done and once updated you can copy your customizations into the new files and amend them as necessary with the word boundaries as described inside each include file.

[mertcangokgoz]

i see this message for new update

nginx: [emerg] unknown directive "~*\badultgalls\b" in /etc/nginx/bots.d/bad-referrer-words.conf:52
nginx: configuration file /etc/nginx/nginx.conf test failed

nginx version: nginx/1.10.3 (Ubuntu)

[kronflux]

This is a local issue, not an issue with the current update, as I'm running it with no problems.

Do you use any custom rules? If so, please post your bad-referrer-words.conf to https://gist.github.com and post the link here. If not, you may want to try re-downloading the file.

I'm not sure if it matters, but also note that your nginx version is quite out of date. we're up to 1.13.5 at this point. It -shouldn't- make a difference, but it's always a good idea for security purposes to update from time to time :)

[mertcangokgoz]

I do not think it's a local problem. I did the update and still the problem continues.

https://gist.github.com/MertcanGokgoz/5d34a10be4953ef3258d3937cd76d9cd

root@local:/etc/nginx# nginx -v nginx version: nginx/1.13.5

[kronflux]

Unfortunate that it had nothing to do with the nginx update.

I'm afraid I have no idea personally, but maybe @mitchellkrogza will have some insight. I can confirm it's definitely not an issue with this recent update, however. I just did a fresh install in a virtual machine of Ubuntu 16.04, installed nginx, and installed the ultimate bad bot blocker. Did a nginx -t, which came back with no issues, and did the curl test to check whether bad referrers were being blocked, and they still are.

So it's definitely an issue with your local setup.

[mitchellkrogza]

@MertcanGokgoz apologies for any problems. It has been tested thoroughly before pushing out the new version. Please can you post your output of nginx -V

For now simply edit bad-referrer-words.conf and delete lines 52 > 66 completely and then do an nginx -t and report back here.

[mitchellkrogza]

@MertcanGokgoz any feedback for me yet? Must isolate the issue so if necessary I can introduce a special test into the TravisCI builds to force it to test on 1.10.x

[mertcangokgoz]

Hello

I deleted the lines from 52 to 66, I tested the problem does not appear

nginx -V output

root@local:~# nginx -V
nginx version: nginx/1.13.5
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

I updated nginx yesterday

[mitchellkrogza]

Thanks @MertcanGokgoz I will check what version we are testing with on TravisCI. I think it is 1.13.2 but I may be mistaken. Will check this and test. At least for now you are up and running again?

[mitchellkrogza]

Ok @MertcanGokgoz I just checked and our TravisCI tests use 1.12.1. I will have to add an additional test into the build process to force download of Mainstream and run a final test with 1.13.x to try and debug why that happened.

[mertcangokgoz]

Thank you for the answer. I use it right now and it does not seem to be a problem, There may be a problem with RegExp but I am not sure, I do not know :)

Good luck with

[mitchellkrogza]

Thanks for reporting it, important that we keep up to date with future versions. I will add an additional test later today into the build process to force download of mainstream and see if we can reproduce the error. It would be strange if Nginx made changes to their Regex interpreter but not totally impossible.

[mitchellkrogza]

@MertcanGokgoz just a quick question, when you pulled the latest include files did you pull the raw versions? ie.

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/custom-bad-referrers.conf -O /etc/nginx/bots.d/custom-bad-referrers.conf

[mertcangokgoz]

error again

root@local:/etc/nginx/sites-available# nginx -t
nginx: [emerg] unknown directive "~*\bsomeveryveryrandomwebsitenamethatdoesnotexist1\.com\b" in /etc/nginx/bots.d/custom-bad-referrers.conf:40
nginx: configuration file /etc/nginx/nginx.conf test failed

[mitchellkrogza]

Ok thanks for testing that again. I am busy implementing an Nginx Mainstream test.

[mertcangokgoz]

thank you. I expect the problem to be resolved,

have a nice day

[mitchellkrogza]

Step 1 complete, we now have nginx/1.13.3 installed on the final test. https://travis-ci.org/mitchellkrogza/nginx-ultimate-bad-bot-blocker/builds/282704354

Now to implement the rest of the testing commands for the blocker on 1.13.3

[mitchellkrogza]

I should have an answer about this by tomorrow, unfortunately summer is here and so are the thunderstorms so in an hour or two we have to run around unplugging everything especially our internet 😢

[mitchellkrogza]

I’m not seeing any such errors yet on Nginx 1.13.3 yet. Not finished yet with my testing but for now let me know what distribution of Linux you are running on so I can test if it’s something specific to your distro.

[mertcangokgoz]

I am using ubuntu and you can follow this details

root@local:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

[kronflux]

is it 32 or 64 bit? and can you also please provide your nginx.conf file with any sensitive info you might have added removed?

I have a strong feeling it's one of the nginx config files. could even be one of the vhost configs, to be honest, since I can't reproduce it myself.

[mertcangokgoz]

I use 64 bit like everyone else, I do not give details about the configuration but i am using https://mozilla.github.io/server-side-tls/ssl-config-generator/ and there are no other additional settings.

[mitchellkrogza]

The nginx.conf file would indeed help isolate this and also if you are running 32/64 bit. All the TravisCI tests are run on Ubuntu 14.04 but all my own servers are running Ubuntu 16.04.2 LTS and none of them are showing this behavior. As a test can you try this too?

sudo apt-get install dos2unix

sudo dos2unix /etc/nginx/conf.d/*

sudo dos2unix /etc/nginx/bots.d/*

[mitchellkrogza]

There should not be anything sensitive in your nginx.conf and if so just remove it

[mitchellkrogza]

And yes to what @kronflux said it could be one of your vhosts that’s causing it. Check them all the make sure the includes are all in the right place.

[mertcangokgoz]

error again, my configuration file does not have an additional setting

root@local:/usr/sbin# sudo dos2unix /etc/nginx/conf.d/*
dos2unix: converting file /etc/nginx/conf.d/botblocker-nginx-settings.conf to Unix format ...
dos2unix: converting file /etc/nginx/conf.d/globalblacklist.conf to Unix format ...
root@local:/usr/sbin# sudo dos2unix /etc/nginx/bots.d/*
dos2unix: converting file /etc/nginx/bots.d/bad-referrer-words.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/blacklist-domains.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/blacklist-ips.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/blacklist-user-agents.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/blockbots.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/custom-bad-referrers.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/ddos.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/whitelist-domains.conf to Unix format ...
dos2unix: converting file /etc/nginx/bots.d/whitelist-ips.conf to Unix format ...
root@local:/usr/sbin# nginx -t
nginx: [emerg] unknown directive "~*\badultgalls\b" in /etc/nginx/bots.d/bad-referrer-words.conf:52
nginx: configuration file /etc/nginx/nginx.conf test failed
root@local:/usr/sbin# 

[mertcangokgoz]

there is an interesting situation

root@local:/usr/sbin# sudo ./setup-ngxblocker -x -e conf
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
no vhost files in: /etc/nginx/sites-available/*.conf => exiting.

but I use .conf

[mitchellkrogza]

Your vhosts should be in /etc/nginx/sites-available and linked using ln -s to /etc/nginx/sites-enabled that’s the standard for Nginx and vhosts should not be defined in nginx.conf this is why you can’t find your error. If you had your vhosts setup as per the Nginx standards your Nginx EMERG error would point you to the exact vhost where the issue is.

[mitchellkrogza]

Sorry only looked properly at your post now, are those sites linked ? What’s your output of

ls -la /etc/nginx/sites-enabled/

[mertcangokgoz]

the same files @mitchellkrogza

[mitchellkrogza]

Truly very weird situation, please can you post your nginx.conf and remove any sensitive information.

[mertcangokgoz]

root@local:/etc/nginx# cat nginx.conf
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;

events {
        worker_connections 1024;
        use epoll;
        multi_accept on;
}

http {

        # limit the number of connections per single IP
        limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
        limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=10r/s;

        ##
        # Basic Settings
        ##
        charset utf-8;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 10;
        keepalive_requests 100;
        lingering_time 20s;
        lingering_timeout 5s;
        keepalive_disable msie6;
        types_hash_max_size 2048;
        server_tokens off;

        client_body_buffer_size 256k;
        client_body_in_file_only off;
        client_body_timeout 60s;
        client_header_buffer_size 64k;
        client_header_timeout  20s;
        client_max_body_size 100m; 

        open_file_cache          max=2000 inactive=20s;
        open_file_cache_valid    60s;
        open_file_cache_min_uses 5;
        open_file_cache_errors   off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##
        access_log off;
        error_log /var/log/nginx/error.log error;
        rewrite_log on;

        log_format      main    '$remote_addr - $remote_user [$time_local] $request '
                        '"$status" $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                        ' "$connection" "$connection_requests" "$request_time"';

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "MSIE [1-6].(?!.*SV1)";
        gzip_vary on;
        gzip_static on;
        gzip_proxied any;
        gzip_comp_level 5;
        gzip_min_length 1400;
        gzip_buffers 32 8k;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;

        ##
        # Virtual Host Configs
        ##
        include /etc/nginx/sites-enabled/*;
}

[mitchellkrogza]

Let me take a look in the morning with some fresh eyes, already I can spot that the default include

include /etc/nginx/conf.d/*;

Is missing from your nginx.conf but it’s late here in South Africa now so let me take a fresh look in the morning, we will resolve this, I always do.

[itoffshore]

@MertcanGokgoz - try changing line 82 to:

local files="$@"

for some reason find $VHOST_DIR -type f -name "*.$VHOST_EXT" is empty

[mertcangokgoz]

I installed nginx from scratch problem solved 👍 I do not know what the problem is

[mitchellkrogza]

@MertcanGokgoz that’s great news thanks for letting us know.

[mitchellkrogza]

It still spurred me on to add the additional testing of the Blocker on mainstream/development version of Nginx which just adds to the solidity of my TravisCI build testing.

[mitchellkrogza]

Thanks to @MertcanGokgoz I have now included testing of the Bad Bot Blocker on the Mainstream/Development version of Nginx within the TravisCI build environment. The blocker passes all the new tests which means it is now thoroughly tested from Nginx 1.10 > 13.3.x 👍

https://travis-ci.org/mitchellkrogza/nginx-ultimate-bad-bot-blocker/builds/283215615