New "super rate limiting" zone (1r/m) in addition of the current one (2r/s)

[bigrck64]

Hello Mitchell

Since a week, ClaudeBot (UA) is harvesting my server from a lot of public IP. I do not want to completly block it, but I want to heavily rate limited it.

Problem: current limiter setting of 2r/s is not enough restrictive. (because this bot is using 100+ different IP, I have 2r/s allowed for each IP -> this is too high for my server)

Current limiter:

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        }

# Rate limiting will only take effect if on any User-Agents with a value of 2
        limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

Would you be able to introduce a second zone with a more restrictive limit like 1r/m ? It would be perfect if I could use another value to activate this new settings for UA (like 4 ?).

# Super rate limiting will only take effect if on any User-Agents with a value of 4
        limit_conn_zone $bot_iplimit zone=bot3_connlimit:16m;
        limit_req_zone  $bot_iplimit zone=bot3_reqlimitip:16m  rate=1r/m;

Moreover, this new setting could be helpfull for other user like i300320 : https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/184

Thanks for considering this request 👍 And a big thumb up for this project !

[mitchellkrogza]

Hi @bigrck64

I'm a bit tied down with some other work. Can you possibly do a test, by manually modifying your globalblacklist.conf as follows

# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------

    map $bad_bot $bot_iplimit {
    0    "";
    1    "";
    2    $binary_remote_addr;
    4    $binary_remote_addr;
    }

# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------

# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2

    limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4

    limit_conn_zone $bot_iplimit zone=bot4_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=bot4_reqlimitip:16m  rate=1r/s;

Let me know if this works

[bigrck64]

Hi Mitchell

Thanks for quick reply ! It seems to work so far :)

Please just change the 1r/s to 1r/m 👍 limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;

Thanks !

[mitchellkrogza]

Hi Mitchell

Thanks for quick reply ! It seems to work so far :)

Please just change the 1r/s to 1r/m 👍 limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;

Thanks !

@bigrck64 Done, feature added - docs updated and tests added into build tests.

[bigrck64]

Awesome ! 😊

[bigrck64]

Maybe you should change

### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### 4 = super rate limited

to

### 1 = allowed
### 2 = rate limited
### 3 = block completely
### 4 = super rate limited

Because for me, 1 does not perform any rate limited task, no ?

[mitchellkrogza]

@bigrck64 correct 1 essentially does nothing, same as 0 bit too late to change that. Will update docs.

[mitchellkrogza]

Maybe you should change

###   1 = allowed or rate limited less restrictive
###   2 = rate limited more
###   3 = block completely
###   4 = super rate limited

to

###   1 = allowed
###   2 = rate limited
###   3 = block completely
###   4 = super rate limited

Because for me, 1 does not perform any rate limited task, no ?

Have done - but explained what 1 actually is/was for and may still be implemented if anyone wants an even less agressive rate limiting zone.

[bigrck64]

Hum, I download your latest version and I feel that "Claudebot" is going into the rate=2r/s instead of rate=1r/m zone :(

Maybe with 2 or 4 it just send the binary_remote_addr into $bot_iplimit:

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        4    $binary_remote_addr;
        }

Which just finish into

# Rate limiting will only take effect if on any User-Agents with a value of 2

        limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

So the solution may be something like

        limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
        limit_req_zone  $bot_iplimit2 zone=bot4_reqlimitip:16m  rate=1r/m;

and

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        4    "";
        }

        map $bad_bot $bot_iplimit2 {
        0    "";
        1    "";
        2    "";
        4    $binary_remote_addr;
        }

?

[mitchellkrogza]

Please add this manually to your local globalblacklist.conf and see if nginx throws any errors - I'm a bit tied down with other stuff

# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================

# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        4    "";
        }

# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------

        map $bad_bot $bot_iplimit2 {
        0    "";
        1    "";
        2    "";
        4    $binary_remote_addr;
        }
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------

# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2

    limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4

    limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
    limit_req_zone  $bot_iplimit2 zone=bot4_reqlimitip:16m  rate=1r/m;

[bigrck64]

Thanks ! No problem with the new nginx config :) The rate limiting seems fine. (edit: it is not 😢)

I can not run your test-blocker-rate-limiting.sh on my prod server, but maybe you will be able to confirm the super rate limiting is effective with it ?

[bigrck64]

Hello Mitchell

I worked on the problem 😊 So I setup a new server to be able to understand your "test-blocker-rate-limiting.sh" And I improved it with the following change:

1) Add the duration and change from 80 to 60 curl requests with the response code:

truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &

2) Wait for all the curl to finish, then display the duration and the number of response code 200 or 503

curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}

echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait

echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"

if [ $SECONDS -lt 30 ]
then
   echo "${bold}${green}PASSED - ${green}GoogleBot was ${bold}${green}RATE LIMITED"
else
   echo "${bold}${red}FAILED - ${red}GoogleBot was ${bold}${red}NOT RATE LIMITED"
fi
printf "\n"

}

3) Same things for Applebot

truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &

And

curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}

echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait

echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"

if [ $SECONDS -gt 30 ]
then
   echo "${bold}${green}PASSED - ${green}Applebot was ${bold}${green}SUPER RATE LIMITED"
else
   echo "${bold}${red}FAILED - ${red}Applebot was ${bold}${red}NOT SUPER RATE LIMITED"
fi
printf "\n"

Conclusion

This new version of test-blocker-rate-limiting.sh show us the duration time of all curl and the % of request ok/rejected 👍 It also confirm that the current implementation of "super rate limited" have the same result than "rate limited".

TEST WITH 2r/s =

        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;
        limit_req_zone  $bot_iplimit zone=bot4_reqlimitip:16m  rate=2r/s;

RESULT

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49

PASSED - GoogleBot was RATE LIMITED

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 5 seconds
Count the number of 200 (response code) = 10
Count the number of 503 (response code) = 50

FAILED - Applebot was NOT SUPER RATE LIMITED

TEST with 1r/m

        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=1r/m;
        limit_req_zone  $bot_iplimit zone=bot4_reqlimitip:16m  rate=1r/m;

RESULT

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58

PASSED - GoogleBot was RATE LIMITED

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58

PASSED - Applebot was SUPER RATE LIMITED

edit1: make the curl wait 90 sec max to not freeze completly the script with nginx super rate limited (1r/m) edit2: add the total duration of 60 curl tests (BASH native) edit3: slow down from 80 to 60 curl request to not slow the test at 2 minuts with normal "rate limited" settings (2r/s) edit4: change [if] from number of 503 to if duration of replied (more stable)

Latest test-blocker-rate-limiting.sh (v2) : test-blocker-rate-limiting.zip

I will now work into nginx globalblacklist.conf to see if I can find a way to make the "super rate limited" to work along with "rate limited" 😊

[bigrck64]

Hi Mitchell,

So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" 😊

Your modification to globalblacklist.conf was perfect, so we keep that:

# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================

# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        4    "";
        }

# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------

        map $bad_bot $bot_iplimit2 {
        0    "";
        1    "";
        2    "";
        4    $binary_remote_addr;
        }
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------

# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2

    limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4

    limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
    limit_req_zone  $bot_iplimit2 zone=bot4_reqlimitip:16m  rate=1r/m;

We only forgot the bot_iplimit2 declaration into blockbots.conf !

limit_conn bot2_connlimit 10;
limit_req  zone=bot2_reqlimitip burst=10;
limit_conn bot4_connlimit 10;
limit_req  zone=bot4_reqlimitip burst=10;
if ($bad_bot = '3') {
  return 444;

This lead to a working validation with new test-blocker-rate-limiting.sh:

• Googlebot (2r/s) got responses in less than 30 sec (2 sec)
• Applebot (1r/m) got responses in more than 30 sec (90 sec)

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49

PASSED - GoogleBot was RATE LIMITED

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58

PASSED - Applebot was SUPER RATE LIMITED

Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf 😊

[mitchellkrogza]

Hi Mitchell,

So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" 😊

Your modification to globalblacklist.conf was perfect, so we keep that:

# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================

# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------

        map $bad_bot $bot_iplimit {
        0    "";
        1    "";
        2    $binary_remote_addr;
        4    "";
        }

# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------

        map $bad_bot $bot_iplimit2 {
        0    "";
        1    "";
        2    "";
        4    $binary_remote_addr;
        }
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------

# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2

  limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
  limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4

  limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
  limit_req_zone  $bot_iplimit2 zone=bot4_reqlimitip:16m  rate=1r/m;

We only forgot the bot_iplimit2 declaration into blockbots.conf !

limit_conn bot2_connlimit 10;
limit_req  zone=bot2_reqlimitip burst=10;
limit_conn bot4_connlimit 10;
limit_req  zone=bot4_reqlimitip burst=10;
if ($bad_bot = '3') {
  return 444;

This lead to a working validation with new test-blocker-rate-limiting.sh:

• Googlebot (2r/s) got responses in less than 30 sec (2 sec)
• Applebot (1r/m) got responses in more than 30 sec (90 sec)

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49

PASSED - GoogleBot was RATE LIMITED

------------------------------
Wait for all 60 curl to finish
------------------------------

Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58

PASSED - Applebot was SUPER RATE LIMITED

Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf 😊

Done, please grab latest of both files and confirm. Tested on my server and working OK - I still need to reintroduce the BETA tests in this repo which broke some time ago.

By default I have left the new super rate limiting zone commented out in blockbots.conf users can decide to enable themselves. It threw an error during the build so I need to as i said reintroduce the beta tests again which test stuff like this without causing any potential breaking changes.

[bigrck64]

Hi Mitchell,

Thanks for the patch to the master branch ! I uninstall / reinstall UBBB to my test and production server (Linux Debian), and it's working as expected, nice ! 👍

I have no error with the uncommented following lines in blockbots.conf, but it's fine like this, user can decide themself to active the "super rate limiter" zone or not 😊

# Uncomment below lines for super rate limiting feature
limit_conn bot4_connlimit 10;
limit_req  zone=bot4_reqlimitip burst=10;

Thanks for all !