Open geerlingguy opened 2 years ago
This took quite some time on my 4 GB access log file, but:
$ awk '{print $1}' access.log | sort | uniq -c | sort -nr
11326588 164.90.211.4
408590 162.158.62.94
386731 162.158.62.96
376055 108.162.219.25
372830 162.158.62.148
276436 127.0.0.1
275066 104.236.203.61
49038 108.162.219.67
40257 139.99.99.165
31197 51.79.144.52
30987 162.158.62.32
30786 173.245.52.202
28335 173.245.52.218
27393 217.16.188.90
25984 128.199.108.29
25235 143.208.200.26
23878 51.159.5.133
22145 108.162.219.129
21551 51.195.188.28
20793 108.162.219.63
20655 133.167.65.45
19872 173.245.52.214
19759 173.245.52.200
18515 133.167.121.133
18133 51.159.3.223
18061 160.16.82.58
17873 108.162.219.177
17848 54.36.176.76
17295 196.192.179.38
17277 159.203.181.211
16608 108.162.219.139
15876 89.40.13.33
15630 134.119.206.106
14842 194.233.69.38
14375 130.185.119.20
14055 35.152.75.76
13848 206.189.151.3
13450 185.199.119.73
13375 47.243.135.104
13344 45.154.255.147
13121 176.9.63.62
13066 92.42.109.186
12490 122.155.165.191
12481 131.255.239.38
12358 160.251.97.210
12283 66.94.120.161
12232 62.205.169.74
12071 103.6.52.35
12009 185.58.19.241
11841 167.86.81.208
11832 139.59.16.95
11799 45.129.56.200
11758 154.126.20.254
11650 200.25.48.72
11464 165.22.216.241
11279 182.253.93.4
11263 185.220.100.254
11254 110.232.67.42
11172 201.20.100.142
11066 209.9.37.60
11042 45.33.26.53
11032 8.215.27.71
11009 204.199.67.174
10835 185.220.100.241
10666 217.196.20.150
10505 92.51.95.194
10456 165.22.26.191
10445 177.129.253.133
10316 104.236.78.102
10038 138.201.120.214
10000 93.104.215.115
I only went to the first bunch that went to 10,000. Would be interesting to do a little IP research and see what all these IPs are doing. The first one looks to be a German DigitalOcean IP address.
I used the 'report abuse' feature on DigitalOcean to report 164.90.211.4
, we'll see what comes of that.
The pattern here suggests DDoS as a service.
Such services leverage compromised hosts from their botnet networks to DDoS victims and each infected host is limited to around 1-5Mbps to avoid detection by their owners/users. So that's 1-5Mbps per host times a few million infected hosts.
So "report abuse" for in these cases, will probably not solve anything. DDoS as a service is growing exponentially every year.
I had a similar issue with my wife's wordpress some weeks ago. I first for i in $(cat logs | cut ... ); do iptables -I INPUT -s ... -j DROP
, but since the patterns were really obvious in the logs (POST requests instead of GET, see below), I ended up with the following fail2ban custom configuration:
$ cat /etc/fail2ban/filter.d/wp-ddos.conf
[Definition]
failregex = ^<HOST> \- \S+ \[\] \"(POST //xmlrpc.php HTTP/1.1|POST /pattern2/ HTTP/1.1|POST /wp-content/ HTTP/1.1)\".+$
$ cat /etc/fail2ban/jail.d/wp-ddos.conf
[wp-ddos]
enabled = true
port = http,https
filter = wp-ddos
logpath = /var/log/nginx/access.log
maxretry = 2
findtime = 60
bantime = 604800
backend = pyinotify
So "report abuse" for in these cases, will probably not solve anything. DDoS as a service is growing exponentially every year.
True, but that one IP was orders of magnitude beyond the rest. Hopefully it will at least prompt DigitalOcean to clean up that server.
A few more quick stats since things are calming down (at least now that Cloudflare's handling the brunt of it:
And looking at my logs from the time before Cloudflare started working... I'm seeing 22,062,150
requests were made in that time frame (about 1-2 hours). On an average day (not just 1-2 hours), my website gets around 1,000,000
HTTP requests.
Can you post your log for the attack? I don't think we need all of it, but I would like to see if there is some sort of fail2ban and/or rate limiting rules that can be done. I mostly just need to see the IP address, time, and request method.
I would also look at Cactus Comments. It is a decentralized comment system based off of Matrix. That will allow you to have comments that don't break through the cache That should prevent some DDOS attacks on your site in the future as well. I don't know how easy it is to integrate into Drupal, and I haven't found a plugin for it, but you know more about how Drupal works then I do.
@minecraftchest1 - Eventually if I find a good comment system that integrates with spam prevention services that I can integrate with a static site and don't feel like it will turn out to be another Discord, I'd love to go that route and convert from Drupal to Hugo or some other static generator.
Well someone is at it again. Approximately 5 minutes ago I got an outage alert, and saw the exact same patterns. Hundreds of thousands of requests like:
150.129.148.88 - - [11/Feb/2022:23:09:04 -0600] "GET https://www.jeffgeerling.com/ HTTP/1.3" 200 11514 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3599.0 Safari/537.36" "-"
...
186.250.160.234 - - [11/Feb/2022:23:09:04 -0600] "POST / HTTP/1.2" 499 0 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; yie11; rv:11.0) like Gecko" "-"
...
At least 150 unique IP addresses and here's a list of the top ones, sampling just a second or two of accesses:
$ awk '{print $1}' access.log | sort | uniq -c | sort -nr
64 185.220.100.244
57 103.120.175.243
56 144.91.85.172
53 78.37.24.148
44 164.70.119.206
40 158.140.170.183
32 45.177.108.162
32 139.228.65.97
31 185.220.101.37
31 103.135.220.91
27 180.246.196.111
22 142.11.199.242
21 201.150.117.164
21 127.0.0.1
19 94.16.15.100
15 104.236.203.61
15 103.175.80.3
14 87.255.13.217
12 88.132.34.230
12 66.94.97.238
11 123.115.115.213
11 119.148.32.25
11 111.68.31.156
11 103.101.17.170
10 165.16.27.13
Cloudflare IPs are popping into the list, since I switched it on again. Wish there were a simple plugin or something for Nginx to just pop on Cloudflare mode and only allow access from those IPs (without me having to pull down the list from Cloudflare itself).
For now, since I'm not on the site and I'm okay with basically disabling comments overnight, I've re-enabled my Cloudflare firewall rule to block all POST requests. The attack is not sophisticated... I might just add a rule in Nginx to block POST requests to anything that's not a valid comment, and see if that holds up (since I don't have the ability to get that deep on Cloudflare).
Seems like the exact same attack that plays out over on Nixcraft: https://www.cyberciti.biz/faq/nginx-block-post-requests-urls-for-spammer-ip-address-cidr/
Here's a fancy bandwidth graph on my server:
This might be of some help. https://stackoverflow.com/questions/47830088/force-cache-on-files-with-nginx I would also see if you can rate limit POST requests. I would also force the use of a cache using the above link. That will make the comments outdated, but will lower the load on your 4G connection.
@minecraftchest1 - Eventually if I find a good comment system that integrates with spam prevention services that I can integrate with a static site and don't feel like it will turn out to be another Discord, I'd love to go that route and convert from Drupal to Hugo or some other static generator.
I don't understand your comment about another Discord.
Since you are using GitHub, you can also look at https://utteranc.es/ as well. Just some options that may help lower attack surfaces.
@minecraftchest1 - I meant Disqus :D — basically, anything that I don't self-host will eventually have an incentive to jam ads and tracking into their product (even if they're a paid service, most likely).
@minecraftchest1 - I meant Disqus :D — basically, anything that I don't self-host will eventually have an incentive to jam ads and tracking into their product (even if they're a paid service, most likely).
That makes more sense. I created #143 to discuss this more as it is off topic for this thread.
Added a note that curl https://www.jeffgeerling.com/blog.xml
returns the correct feed, but some users (notably, for NetNewsWire) are seeing duplicates of every article I post now. Not sure what's going on, but I also asked out loud on Twitter: https://twitter.com/geerlingguy/status/1498438448520253443
Edit: Someone else also noted this in newsboat, so it's not an isolated incident:
Opened new issue for the duplicate RSS feed items: https://github.com/geerlingguy/jeffgeerling-com/issues/145
Another attack after I posted the video How I survived a DDoS attack.
Nginx request log:
Set Cloudflare to 'under attack' mode for now since I have other things to do this morning :P
root@www:/var/log/nginx# awk '{print $1}' access.log | sort | uniq -c | sort -nr
119742 162.158.62.94
108478 162.158.62.96
103952 108.162.219.25
99629 162.158.62.148
24292 127.0.0.1
24100 104.236.203.61
20404 108.162.219.67
11709 173.245.52.202
11559 162.158.62.32
10885 173.245.52.218
10820 108.162.219.63
9829 108.162.219.129
8950 173.245.52.214
8742 162.158.63.61
7416 173.245.52.200
7055 108.162.219.139
6326 108.162.219.177
5168 162.158.62.178
2029 108.162.219.21
1321 162.158.63.59
1269 162.158.63.51
1225 108.162.219.153
1153 108.162.219.107
1089 162.158.62.56
1079 162.158.62.140
1039 173.245.52.206
1033 162.158.62.40
1014 173.245.52.208
Obviously most of the requests are coming through CF—I need to modify my command (will do later) to grab the real IP address.
This time the script is hitting /user/login, which I set to bypass (will have to switch that off, lol):
108.162.219.25 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; Touch; rv:11.0) like Gecko" "66.187.4.202"
127.0.0.1 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.0" 200 8736 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7.1; rv:8.3) Gecko/20100101 Firefox/8.3.6" "162.158.62.148"
162.158.62.148 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7.1; rv:8.3) Gecko/20100101 Firefox/8.3.6" "176.214.34.58"
173.245.52.214 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Windows; U; Windows NT 5.3) AppleWebKit/533.1.0 (KHTML, like Gecko) Chrome/14.0.880.0 Safari/533.1.0" "2604:2dc0:200:1654::"
162.158.62.96 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5.9; rv:9.3) Gecko/20100101 Firefox/9.3.9" "201.149.103.113"
Decided to just block the /user/login
path entirely for now. Nobody but me needs access to it ;)
One of the top IPs seems to belong to an elementary school... https://ipinfo.io/165.139.238.20
Probably a hacked teacher's computer.
Hundreds of thousands of requests blocked now:
Requests are back down to a more normal rate on the backend:
Up to 1.3 million now.
Edit: 1.8 million after another minute. lol
Edit 2: 2.2 million after another minute
@geerlingguy awesome stuff
looks like the site is still working without issues
@geerlingguy report them to OVH and others. the last one is OVH's, their email is abuse@ovh.ca
@geerlingguy
lol, they saw my block on /user/login and now they're targeting /user/password.
104.236.203.61 - - [16/Mar/2022:13:45:33 -0500] "GET /user/password HTTP/1.0" 500 604 "https://www.jeffgeerling.com/user/password" "Mozilla/5.0 (Windows; U; Windows NT 6.3) AppleWebKit/537.1.2 (KHTML, like Gecko) Chrome/16.0.808.0 Safari/537.1.2" "162.158.62.148"
108.162.219.147 - - [16/Mar/2022:13:45:33 -0500] "GET /user/password HTTP/1.1" 500 202 "https://www.jeffgeerling.com/user/password" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7.4; rv:11.0) Gecko/20100101 Firefox/11.0.3" "118.212.51.231"
162.158.62.148 - - [16/Mar/2022:13:45:33 -0500] "GET /user/password HTTP/1.1" 500 604 "https://www.jeffgeerling.com/user/password" "Mozilla/5.0 (Windows; U; Windows NT 6.3) AppleWebKit/537.1.2 (KHTML, like Gecko) Chrome/16.0.808.0 Safari/537.1.2" "2604:2dc0:200:1654::"
Cloudflare's 'Under Attack' mode can keep things alive, but there are still hundreds of thousands of requests coming through.
Now back to none.
If whomever is aiming the DDoS cannons is reading this thread, don't you have anything better to do? I mean, at least target some site that would be beneficial to have off the Internet 🤣
Well, block user/*
completely
(And to give an update on why I didn't fix that sooner—I was out getting lab work done for Crohn's medicine. I have plenty of other stuff in my life that are higher priority than my website 😉)
I have diabetes myself, so I fully understand
@PH4NTOMiki - I might just add a couple more rules in soon and block off anything besides comment forms that isn't a public-facing page.
I have diabetes myself, so I fully understand
I hate when I have to wait for some stuff just to be able to get insulin and insulin pump supplies for myself, so no worries.
All right, here it is sorted by true IPs since I've had a little time without needing to head off to an appointment:
root@www:/var/log/nginx# awk '{print $(NF)}' access.log | sort | uniq -c | sort -nr
639380 "2604:2dc0:200:1654::"
212241 "59.124.224.205"
153780 "172.104.48.219"
123187 "121.240.20.207"
106175 "161.189.131.134"
95478 "52.230.124.4"
93081 "20.212.145.207"
92533 "2001:41d0:403:3ee8::"
91538 "138.91.42.88"
90668 "162.158.62.94"
80397 "2402:f000:4:1005:809:ffff:feab:68f6"
79298 "162.158.62.148"
78999 "108.162.219.25"
78870 "162.158.62.96"
78837 "66.187.4.202"
77513 "66.187.5.165"
75162 "154.206.15.27"
74699 "167.172.239.13"
74402 "132.145.75.227"
74378 "129.153.182.100"
73588 "139.180.210.100"
72775 "139.180.219.217"
68679 "205.144.61.11"
65524 "118.212.51.231"
64987 "66.187.6.143"
63971 "222.218.122.68"
53136 "171.244.5.175"
52549 "134.119.217.106"
48803 "37.61.222.162"
47536 "113.160.210.32"
47358 "190.108.87.242"
46221 "190.8.248.12"
44600 "184.82.48.13"
44524 "114.246.98.18"
42939 "54.233.87.212"
40235 "103.145.149.62"
40153 "112.78.133.138"
38498 "190.104.1.21"
36817 "45.169.136.183"
36259 "105.213.87.102"
36245 "112.78.164.191"
35867 "80.240.126.125"
35564 "139.59.99.117"
34750 "54.233.220.165"
33209 "173.212.213.133"
32823 "117.53.47.183"
31757 "118.99.113.162"
30759 "201.149.103.113"
29875 "54.179.161.157"
29298 "85.195.123.194"
29159 "37.61.220.234"
29018 "54.150.109.80"
28789 "172.107.194.40"
28750 "54.250.88.122"
28113 "36.92.109.146"
26400 "113.195.32.40"
26188 "54.255.103.5"
26049 "27.46.55.25"
25753 "35.78.23.118"
25193 "2a01:4f9:3b:478b::2"
25131 "118.99.115.85"
23899 "45.189.252.130"
23557 "160.251.49.219"
22894 "45.184.130.179"
21845 "45.230.47.30"
21837 "18.216.136.190"
20916 "190.120.250.220"
20813 "181.78.20.45"
20677 "103.17.246.149"
20557 "125.162.41.84"
20415 "171.4.69.50"
20147 "182.72.234.138"
20107 "182.253.158.136"
19832 "183.89.194.236"
18591 "209.145.35.79"
18260 "182.253.47.89"
18096 "190.60.71.124"
18016 "52.194.231.209"
17902 "181.78.1.219"
17642 "66.187.6.119"
17109 "105.213.0.55"
17055 "134.209.246.160"
16054 "102.64.66.106"
15427 "72.80.241.81"
15204 "118.212.49.107"
14116 "222.67.185.94"
14083 "108.162.219.67"
13960 "118.99.108.74"
13742 "176.214.34.58"
13524 "173.245.52.202"
13509 "138.122.6.139"
13502 "160.154.123.227"
13158 "162.158.62.32"
13064 "181.49.231.246"
12891 "173.245.52.218"
12291 "138.68.81.255"
12219 "103.14.20.28"
12067 "108.162.219.63"
11975 "118.99.111.234"
11418 "47.243.187.173"
10942 "13.59.34.132"
10765 "95.71.49.87"
10403 "139.99.88.19"
That's just IPs that sent more than 10,000 requests in the past couple hours.
Up to 7,000,000 blocked today on Cloudflare (and a few million more that went through):
Just a few more graphs from while I was getting my labs drawn at the hospital:
lol, back at it again:
162.158.62.96 - - [16/Mar/2022:14:33:12 -0500] "GET / HTTP/1.1" 502 182 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 5.3; rv:9.2) Gecko/20100101 Firefox/9.2.8" "188.214.129.185,165.139.238.20"
108.162.219.129 - - [16/Mar/2022:14:33:12 -0500] "GET / HTTP/1.1" 502 584 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.3) AppleWebKit/538.0.0 (KHTML, like Gecko) Chrome/28.0.809.0 Safari/538.0.0" "190.107.224.150"
108.162.219.25 - - [16/Mar/2022:14:33:12 -0500] "GET / HTTP/1.1" 502 584 "https://www.jeffgeerling.com/" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.3; Trident/3.1; .NET CLR 1.2.86301.7)" "181.205.41.210"
108.162.219.25 - - [16/Mar/2022:14:33:12 -0500] "GET / HTTP/1.1" 502 584 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.3) AppleWebKit/531.1.1 (KHTML, like Gecko) Chrome/28.0.842.0 Safari/531.1.1" "2001:19f0:6001:10ee:7633:4121:b90c:cd2c"
108.162.219.25 - - [16/Mar/2022:14:33:12 -0500] "GET / HTTP/1.1" 502 182 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 5.3; WOW64; rv:6.1) Gecko/20100101 Firefox/6.1.7" "121.240.20.207"
I guess your only option is to switch to some static site generator (I suggest Hexo) when you have time
I can help with the migration if you want
https://www.drupal.org/project/static_generator appears to be able to generate a static version of a drupal site that could be deployed to cloudflare/github/gitlab pages, or served via ipfs using an ipfs gateway.
If I stuck on Drupal, I'd probably use Tome (https://www.drupal.org/project/tome). But for other projects I either use Jekyll or Hugo :)
So the attacker finally became wise enough to start posting comments, and in a stroke of actual intelligence, decided to use the edit domain.
I left that route open to see whether the attacker would eventually make it through or not, and it looks like he or she did.
So this morning I:
It seems like some requests are still squeaking through DO's firewall rule, though—for example:
108.162.219.180 - - [17/Mar/2022:09:47:38 -0500] "POST /comment/reply/node/3192/comment_node_blog_post HTTP/1.1" 403 206 "https://edit.jeffgeerling.com/comment/reply/node/3192/comment_node_blog_post" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" "127.0.0.1, 127.0.0.1,65.74.147.27"
162.158.63.14 - - [17/Mar/2022:09:47:38 -0500] "POST /comment/reply/node/3192/comment_node_blog_post HTTP/1.1" 403 206 "https://edit.jeffgeerling.com/comment/reply/node/3192/comment_node_blog_post" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" "127.0.0.1, 127.0.0.1,65.74.147.27"
162.158.63.242 - - [17/Mar/2022:09:47:38 -0500] "POST /comment/reply/node/3192/comment_node_blog_post HTTP/1.1" 403 206 "https://edit.jeffgeerling.com/comment/reply/node/3192/comment_node_blog_post" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" "127.0.0.1, 127.0.0.1,65.74.147.27"
108.162.219.60 - - [17/Mar/2022:09:47:38 -0500] "POST /comment/reply/node/3192/comment_node_blog_post HTTP/1.1" 403 206 "https://edit.jeffgeerling.com/comment/reply/node/3192/comment_node_blog_post" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" "127.0.0.1, 127.0.0.1,65.74.147.27"
108.162.219.60 - - [17/Mar/2022:09:47:39 -0500] "POST /comment/reply/node/3192/comment_node_blog_post HTTP/1.1" 403 206 "https://edit.jeffgeerling.com/comment/reply/node/3192/comment_node_blog_post" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" "127.0.0.1, 127.0.0.1,65.74.147.27"
Maybe it's just queued requests.
Will have to clean up these comments later:
Comment reply blocking rule was working for a while:
But now DigitalOcean seems to be tiring of handling this idiot's barrage:
I may move to another server, and drop all DNS routes to the server itself.
At least they recommend a service you use.
I ended up basically dropping the IP associated with the backend services and am on a new IP at this point, with full proxy coverage on Cloudflare so the attacker can't get around it.
I also set Cloudflare's security level 'high' for now (I'll back that off once things stabilize), but in some browsers I'm getting a 'Too many redirects' warning. Not exactly sure what's going on there, but I just rebooted the web server.
I'm wondering if by changing IP addresses, Nginx may have somehow gotten confused about what IP it was serving from, and tried redirecting through the old IP? Not sure, we'll see.
That might've been it. Before the reboot, requests were coming through like:
127.0.0.1 - - [17/Mar/2022:12:51:25 -0500] "GET /tags/youtube?page=3 HTTP/1.0" 200 34511 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" "108.162.219.144"
127.0.0.1 - - [17/Mar/2022:12:51:27 -0500] "GET /photos/2010/2106 HTTP/1.0" 200 8124 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "173.245.52.195"
And after the reboot:
162.158.62.151 - - [17/Mar/2022:13:00:11 -0500] "GET /taxonomy/term/700/feed HTTP/1.1" 200 2469 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" "185.191.171.22"
162.158.63.64 - - [17/Mar/2022:13:00:13 -0500] "GET /comment/17696 HTTP/1.1" 200 26626 "-" "Mozilla/5.0 (compatible; Linespider/1.1; +https://lin.ee/4dwXkTH)" "125.209.235.167"
108.162.219.48 - - [17/Mar/2022:13:00:15 -0500] "GET /blog/2016/latest-diy-project-fireplacechimney-removal-nicer-living-room HTTP/1.1" 200 5089 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" "51.222.253.1"
108.162.219.110 - - [17/Mar/2022:13:00:27 -0500] "GET /tags/access-control HTTP/1.1" 200 3414 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" "51.222.253.14"
Probably could've just restarted Nginx itself.
Also heard from someone running some security honeypots that there were a few other amplification attacks that were attempted. Maybe Putin has nothing better to do since his army is kinda stalled out, and he's assigned someone to try taking down my website since I mentioned something about supporting Ukraine 🤣.
So, shortly after my post Hosting this website on a farm - or anywhere went live, the site was pummeled by an average of 5,000,000 POST requests per hour (in addition to a ton of other traffic, I don't even have the full metrics because I turned some of that off once the server load was hitting 25-30).
There was no way the poor Turing Pi 2 cluster could handle that load—and not even my 2GB DigitalOcean VPS could.
So I had to turn to Cloudflare. I went 13 years without needing a hosted CDN/cache layer in front of my site, and went through dozens of HN front page sessions without an issue.
But when someone points a DDoS cannon at the site (I could confirm at least 1,000 unique IPs were sending dozens of requests per second), requiring Cloudflare is inevitable.
A few things I need to clean up after all this insanity:
curl https://www.jeffgeerling.com/blog.xml
, so I'm not sure what's happening for these users. (see https://github.com/geerlingguy/jeffgeerling-com/issues/145)