[BUG] HTTP response code 444 causing 502 on AWS ELB

[hong823]

Describe the bug

Currently, the bad bot blocker is responding HTTP status code 444 when it detects a bad bot. However, when nginx was hosted behind an AWS ELB it was responded with HTTP status code 502

I was debugging this with AWS and they told me that AWS ELB could not recognize HTTP status code 444 as it was a non-standard HTTP status code and responded with HTTP status code 502

The visualized flow current looks like this:

Nginx (Respond 444) -> AWS ELB (Respond 502) -> Bad Bots (Received 502)

Below are one of the example configuration files in /etc/nginx/bots.d/blockbots.conf that return status code 444:

if ($bad_bot = '3') {
  return 444;
  }

# ---------------------
# BLOCK BAD REFER WORDS
# ---------------------

if ($bad_words) {
  return 444;
}

# ------------------
# BLOCK BAD REFERERS
# ------------------

if ($bad_referer) {
  return 444;
}

# -----------------------------
# BLOCK IP ADDRESSES and RANGES
# -----------------------------

if ($validate_client) {
  return 444;
}

To Reproduce

Steps to reproduce the behavior:

1. Host an nginx behind AWS ELB (Either on EC2, ECS, or EKS) with this bad bot blocker package installed
2. Execute the bottom command:

curl -A "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" -I https://your-aws-elb-url.com

3. Noticed the response from above was responded with HTTP status code 502

   HTTP/2 502
   date: Tue, 01 Dec 2020 20:11:28 GMT
   content-type: text/html; charset=UTF-8
   content-length: 122
   server: awselb/2.0

Expected behavior

I would expect that either the response code on bad blocker would be configurable (Not sure how possible is this) or at least responded with HTTP status code 403 so that AWS ELB and interpreted it correctly and responded with HTTP status code 403 as well.

The visualized flow should looks like this: Nginx (Respond 403) -> AWS ELB (Respond 403) -> Bad Bots (Received 403)

Copy of nginx.conf

http {
    ...

    include /etc/nginx/conf.d/botblocker-nginx-settings.conf;
    include /etc/nginx/conf.d/globalblacklist.conf;

    ...
}

Copy of vhost / website / host .conf file

server {
    ...

    include /etc/nginx/bots.d/ddos.conf; 
    include /etc/nginx/bots.d/blockbots.conf;

    ...
}

Server (please complete the following information):

• Operating System:

• [ ] Ubuntu

• [x] Alpine

• [ ] Arch Linux

• [ ] Debian

• [ ] CentOS

• [ ] Fedora

• [ ] Deepin

• [ ] Windows

• [ ] Other

• Specify Exact Version of OS: Alpine Linux v3.9.4

Post output of uname -a here

Linux common-nginx-7fbdd8cb8c-xvl88 4.14.203-156.332.amzn2.x86_64 #1 SMP Fri Oct 30 19:19:33 UTC 2020 x86_64 Linux

• Nginx Version [post output of sudo nginx -v]

nginx version: nginx/1.16.0

• Other Environments [include Version information]

• [ ] Plesk

• [ ] CPanel

• [ ] Synology NAS

• [ ] Other

• Specify Other / Specific Version Information Here:

• Any other applicable log / error messages that may help us to help you.

Nginx request logs

{
  "remote_user": "",
  "body_bytes_sent": "0",
  "request_time": "0.000",
  "http_referrer": "",
  "http_user_agent": "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)",
  "request_uri": "xxx",
  "request_method": "GET",
  "status": "444",
  "server_port": "443",
  "server_protocol": "HTTP/1.1",
  "ssl_protocol": "TLSv1.2",
}

Additional information

Add any other context about the problem here.

[hong823]

Another alternative I was thinking about is to have a copy of /etc/nginx/bots.d/blockbots.conf which overwrite return 444; into return 403;. But I'm not sure if that is good practice to do for a temporary fix?

[mitchellkrogza]

blockbots.conf and the response code you choose to issue has been 100% user configurable since day 1 and has never changed. ANY modifications to blockbots.conf are NOT over-written during an update.

[mitchellkrogza]

@hong823 blockbots.conf has been update with comments to reflect this.

[hong823]

@mitchellkrogza thanks for the clarification!

[hong823]

@mitchellkrogza I noticed some of the configurable files are not mentioned for /etc/nginx/bots.d in the instructions, I've created #410 to include them.