search

SvarDOS / bugz

SvarDOS bug tracker
http://svardos.org/
7 stars 0 forks source link

svardos.org inaccessible #137

Open boeckmann opened 2 weeks ago

boeckmann commented 2 weeks ago

I currently cannot access svardos.org. Is maintenance going on?

mateuszviste commented 2 weeks ago

image

apparently the load is caused again by a facebook bot hitting on the svn web frontend...

57.141.0.9 - - [12/Nov/2024:13:48:08 +0000] "GET /filedetails.php?repname=SvarDOS&path=%2Finstall%2F&rev=49 HTTP/1.1" 404 5408 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.7 - - [12/Nov/2024:13:48:08 +0000] "GET /listing.php?repname=SvarDOS&path=%2Fsvarcom%2Ftags%2F&rev=474&peg=1090 HTTP/1.1" 200 18639 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.18 - - [12/Nov/2024:13:48:00 +0000] "GET /diff.php?repname=SvarDOS&path=%2Finstall%2Ftrunk%2Fnls%2Fsv_utf8.txt&rev=1946&peg=857 HTTP/1.1" 200 12674 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.23 - - [12/Nov/2024:13:48:09 +0000] "GET /revision.php?repname=SvarDOS&path=%2Fsvarcom%2Ftrunk%2F&rev=1952&peg=592 HTTP/1.1" 200 9223 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.30 - - [12/Nov/2024:13:48:01 +0000] "GET /diff.php?repname=SvarDOS&path=%2Fhelp%2Fhelp-de%2F&rev=1819&peg=1386 HTTP/1.1" 200 8186 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.29 - - [12/Nov/2024:13:48:10 +0000] "GET /blame.php?repname=SvarDOS&path=%2Flocalcfg%2F&rev=594&peg=1317 HTTP/1.1" 200 7677 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.15 - - [12/Nov/2024:13:48:15 +0000] "GET /rss.php?repname=SvarDOS&path=%2Fpkg%2Ftrunk%2Ftrim.c&peg=1315 HTTP/1.1" 200 8229 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.12 - - [12/Nov/2024:13:48:12 +0000] "GET /log.php?repname=SvarDOS&path=%2Fwebsite%2Findex-why.htm&rev=137&peg=220 HTTP/1.1" 200 11786 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.23 - - [12/Nov/2024:13:48:13 +0000] "GET /log.php?repname=SvarDOS&path=%2F&rev=1164&peg=1197&isdir=1&page=2&showchanges=1 HTTP/1.1" 200 28098 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.7 - - [12/Nov/2024:13:48:05 +0000] "GET /blame.php?repname=SvarDOS&path=%2Fpkgnet%2Ftrunk%2Fwatt32%2Finc%2Fnetinet6%2Fin6.h&peg=676 HTTP/1.1" 200 199079 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.6 - - [12/Nov/2024:13:48:06 +0000] "GET /listing.php?repname=SvarDOS&path=%2Ffiles%2F&rev=311&peg=602 HTTP/1.1" 200 10782 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.24 - - [12/Nov/2024:13:48:12 +0000] "GET /revision.php?repname=SvarDOS&path=%2Fhelp%2Fhelp-de%2Findex.ama&rev=698&peg=1395 HTTP/1.1" 200 10373 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
34.233.219.155 - - [12/Nov/2024:13:48:07 +0000] "GET /listing.php?path=//&peg=75&repname=SvarDOS&rev=35 HTTP/1.1" 200 3608 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
52.45.92.83 - - [12/Nov/2024:13:48:07 +0000] "GET /listing.php?path=/&peg=75&repname=SvarDOS&rev=577 HTTP/1.1" 200 4093 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
57.141.0.11 - - [12/Nov/2024:13:47:53 +0000] "GET /listing.php?repname=SvarDOS&path=%2Fpackages%2F&rev=880&peg=884 HTTP/1.1" 200 674991 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.18 - - [12/Nov/2024:13:48:13 +0000] "GET /log.php?repname=SvarDOS&path=%2Flocalcfg%2F&isdir=1&rev=1162&all=1 HTTP/1.1" 200 28280 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
57.141.0.11 - - [12/Nov/2024:13:48:02 +0000] "GET /diff.php?repname=SvarDOS&path=%2Fpkgnet%2Ftrunk%2Fnet.c&rev=1132&peg=1111 HTTP/1.1" 200 25721 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
mateuszviste commented 2 weeks ago

disabled the svn.svardos.org websvn frontend. system stabilized immediately. maybe time to revisit #22

boeckmann commented 2 weeks ago

Yes, for me it is working again :)

mateuszviste commented 2 weeks ago

This same issue occurred a couple of months ago - "same" meaning that facebook was hitting so hard on the websvn frontend that there was no cpu left for anything else. Back then I had added a htaccess blacklist to reject facebook:

root@vps-0e3e8454:/srv/svn.svardos.org_websvn/websvn-2.8.3# cat .htaccess 
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^facebookexternal.* [NC]
RewriteRule ^ - [F,L]
root@vps-0e3e8454:/srv/svn.svardos.org_websvn/websvn-2.8.3#

From today's logs I see that facebook changed the user-agent string of their insane bot and now it presents as "meta-externalagent" instead of "facebookexternal"...

I have changed the htaccess to this:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} .*facebook.* [NC]
RewriteRule ^ - [F,L]

Seems to work.

I have also added a robot.txt. We'll see.

mateuszviste commented 2 weeks ago

There is also a thing called "amazonbot" that constantly fetches stuff on svn.svardos.org, but this one is behaved enough to rate-limit itself to about 1 query per second.

52.203.65.83 - - [12/Nov/2024:14:48:13 +0000] "GET /log.php?path=//&peg=75&repname=SvarDOS&rev=462 HTTP/1.1" 200 4276 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
44.193.115.232 - - [12/Nov/2024:14:48:14 +0000] "GET /log.php?path=//&peg=73&repname=SvarDOS&rev=698 HTTP/1.1" 200 4166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
34.192.125.239 - - [12/Nov/2024:14:48:15 +0000] "GET /log.php?path=//&peg=72&repname=SvarDOS&rev=750 HTTP/1.1" 200 4186 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
34.233.114.237 - - [12/Nov/2024:14:48:16 +0000] "GET /log.php?path=//&peg=73&repname=SvarDOS&rev=697 HTTP/1.1" 200 4175 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
23.22.105.143 - - [12/Nov/2024:14:48:17 +0000] "GET /log.php?path=//&peg=75&repname=SvarDOS&rev=461 HTTP/1.1" 200 4231 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
52.3.127.170 - - [12/Nov/2024:14:48:18 +0000] "GET /log.php?path=//&peg=55&repname=SvarDOS&rev=963 HTTP/1.1" 200 4303 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
54.84.169.196 - - [12/Nov/2024:14:48:19 +0000] "GET /log.php?path=//&peg=54&repname=SvarDOS&rev=2172 HTTP/1.1" 200 4406 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
98.82.40.168 - - [12/Nov/2024:14:48:20 +0000] "GET /log.php?path=//&peg=55&repname=SvarDOS&rev=962 HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
54.91.122.193 - - [12/Nov/2024:14:48:21 +0000] "GET /log.php?path=//&peg=75&repname=SvarDOS&rev=460 HTTP/1.1" 200 4248 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
44.209.187.99 - - [12/Nov/2024:14:48:22 +0000] "GET /log.php?path=//&peg=73&repname=SvarDOS&rev=696 HTTP/1.1" 200 4160 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
52.6.232.201 - - [12/Nov/2024:14:48:23 +0000] "GET /log.php?path=//&peg=72&repname=SvarDOS&rev=749 HTTP/1.1" 200 4204 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
3.221.244.28 - - [12/Nov/2024:14:48:24 +0000] "GET /log.php?path=//&peg=73&repname=SvarDOS&rev=695 HTTP/1.1" 200 4139 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
3.216.86.144 - - [12/Nov/2024:14:48:25 +0000] "GET /blame.php?path=//&peg=55&repname=SvarDOS&rev=2228 HTTP/1.1" 200 2619 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
40.77.167.8 - - [12/Nov/2024:14:48:26 +0000] "GET /filedetails.php?repname=SvarDOS%20Packages&path=%2Fcrc32-20040215.zip HTTP/1.1" 200 3334 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"
35.168.238.50 - - [12/Nov/2024:14:48:26 +0000] "GET /diff.php?path=//&peg=55&repname=SvarDOS&rev=2228 HTTP/1.1" 200 2591 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
3.224.104.67 - - [12/Nov/2024:14:48:27 +0000] "GET /filedetails.php?path=//&peg=55&repname=SvarDOS&rev=2228 HTTP/1.1" 404 5456 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"