search

SvarDOS / bugz

SvarDOS bug tracker
http://svardos.org/
7 stars 0 forks source link

SvarDOS Forum is broken! #57

Closed bttrx closed 10 months ago

bttrx commented 10 months ago

Hi @mateuszviste!

Today I noticed the following problems at http://svardos.org/?p=forum

  1. Creating a new thread gives me ERROR: Failed to create thread #1705871039 (BŁĄD: nie zdołano utworzyć wątku nr 1705871039).
  2. Searching for minibb returns no result, but a search for fdisk works.
  3. Clicking one of the fdisk hits directs me to the start page of the forum, but not to the thread.
  4. I left a test answer at http://svardos.org/?p=forum&thread=1705432205#1705871596, but fields 'author', 'address', 'digital signature', and 'message' show up blank.
mateuszviste commented 10 months ago

my VPS' storage space was 100% full... I removed a few things, should be good for the time being. Will look into a better solution on the long run.

bttrx commented 10 months ago

Did we kill it again by uploading JEMM 5.83? (@boeckmann)

http://svn.svardos.org/diff.php?repname=SvarDOS&path=%2Fhelp%2Fhelp.bat&rev=1650&peg=1650 returns: Fehler beim Ausführen des Befehls: diff -U 5 "/tmp/jcHgnI" "/tmp/JVIL4G" No output on STDOUT.

mateuszviste commented 10 months ago

the issue appears to come from this insanity:

root@vps-0e3e8454:/tmp# du -ksh /tmp/*
26G     /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MACHxh
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-chrony.service-n9q9wf
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-systemd-logind.service-suowxi
root@vps-0e3e8454:/tmp#
root@vps-0e3e8454:/tmp# ll /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MACHxh/tmp/ | head
total 26859616
-rw------- 1 www-data www-data        0 Jan 22 18:29 004tOE
-rw-r--r-- 1 www-data www-data        0 Jan 22 18:29 004tOEhighlight
-rw------- 1 www-data www-data     1973 Jan 20 19:34 006bbi
-rw-r--r-- 1 www-data www-data     2016 Jan 20 19:34 006bbihighlight
-rw------- 1 www-data www-data      217 Jan  3 18:29 00AkUA
-rw-r--r-- 1 www-data www-data      672 Jan  3 18:29 00AkUAhighlight
-rw------- 1 www-data www-data        0 Jan 22 23:11 00EZiY
-rw-r--r-- 1 www-data www-data        0 Jan 22 23:11 00EZiYhighlight
-rw------- 1 www-data www-data   134128 Dec 22 19:43 00GtFP
root@vps-0e3e8454:/tmp# 
root@vps-0e3e8454:/tmp# ll /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MACHxh/tmp/ | tail
-rw------- 1 www-data www-data  6590301 Jan 21 06:08 zzrV6U
-rw-r--r-- 1 www-data www-data  7174725 Jan 21 06:08 zzrV6Uhighlight
-rw------- 1 www-data www-data     3008 Dec 15 11:25 zzrXAx
-rw-r--r-- 1 www-data www-data     3070 Dec 15 11:25 zzrXAxhighlight
-rw------- 1 www-data www-data      111 Jan 11 19:26 zzuK9b
-rw-r--r-- 1 www-data www-data      112 Jan 11 19:26 zzuK9bhighlight
-rw------- 1 www-data www-data     2907 Jan  1 13:57 zzxSUM
-rw-r--r-- 1 www-data www-data     6312 Jan  1 13:57 zzxSUMhighlight
-rw------- 1 www-data www-data     3223 Dec 28 13:07 zzytPn
-rw-r--r-- 1 www-data www-data     3244 Dec 28 13:07 zzytPnhighlight
root@vps-0e3e8454:/tmp# ll /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MACHxh/tmp/ | wc -l
213724
root@vps-0e3e8454:/tmp#

I never liked systemd, but now I like it even less. And I do not know why systemd creates such huge tmp directory.

I have restarted the apache2 process and suddenly systemd came back to reason:

root@vps-0e3e8454:/# du -ksh /tmp/*
244K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-chrony.service-n9q9wf
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-systemd-logind.service-suowxi
root@vps-0e3e8454:/#

I guess I will have to restart the thing every now and then through some cron. The uptime of this server is quite impressive BTW:

root@vps-0e3e8454:/# uptime
 04:19:58 up 786 days, 13:44,  1 user,  load average: 0.13, 0.14, 0.07
root@vps-0e3e8454:/#
mateuszviste commented 10 months ago

after only a few minutes the systemd-tmp thing is over 4MiB already.

root@vps-0e3e8454:/# du -ksh /tmp/*
4.0K    /tmp/ogup
4.0M    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-chrony.service-n9q9wf
8.0K    /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-systemd-logind.service-suowxi
root@vps-0e3e8454:/# ss -t
State            Recv-Q            Send-Q                                 Local Address:Port                                     Peer Address:Port             Process            
ESTAB            0                 52                                     141.95.149.60:ssh                                   176.138.251.144:35944                               
ESTAB            0                 0                             [::ffff:141.95.149.60]:http                         [::ffff:111.225.149.149]:23146                               
ESTAB            0                 0                             [::ffff:141.95.149.60]:http                          [::ffff:65.108.227.178]:58650                               
root@vps-0e3e8454:/#

I thought it could be related to some malicious third party opening lots of connections and keeping them idle, but apparently not.

mateuszviste commented 10 months ago

Looking inside these weird tmp files I see they contain chunks of SvarDOS source code, hence I am suspecting that it is somehow caused by the websvn frontend running at http://svn.svardos.org.

root@vps-0e3e8454:/tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg/tmp# cat SG3amv | head
#!/bin/sh
#
# This script generates indexes of Svarog386 repositories and builds the ISO
# CD images. It should be executed each time that a package file have been
# modified, added or removed from any of the repositories.
#

### parameters block starts here ############################################

REPOROOT=/srv/www/svarog386.viste.fr/repos/
root@vps-0e3e8454:/tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg/tmp# cat M11QKy
/*
 * fileexists checks whether a file exists or not.
 * returns 0 if file not found, non-zero otherwise.
 */

#ifndef fileexists_sentinel
#define fileexists_sentinel

int fileexists(const char *filename);

#endif
root@vps-0e3e8454:/tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg/tmp# ^[[200~cat wKoBVT
-bash: $'\E[200~cat': command not found
root@vps-0e3e8454:/tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg/tmp# cat wKoBVT
/*
 * This file is part of the SvarDOS project.
 * Copyright (C) Mateusz Viste 2012-2021
 */

#ifndef pkgrem_sentinel
#define pkgrem_sentinel
int pkgrem(const char *pkgname, const char *dosdir);
#endif
root@vps-0e3e8454:/tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-JWuCYg/tmp#
mateuszviste commented 10 months ago

I have disabled svn.svardos.org for the time being. the /tmp directory on the server ceased to grow, so the culprit is identified.

Removing the service is not a very good solution, though, as it is a pretty useful thing... Some alternative would have to be found.

In #22 Roy suggested leveraging the github platform for this. In lack of a better alternative it is probably not a bad option.

mateuszviste commented 10 months ago

Just for extra context: it seems that websvn "forgets" to clean up some of its TMP file and systemd also leaves them pending around, which quickly builds up. This was apparently triggered by some "AmazonBot" spider that seems to be intensively scanning svn.svardos.org.

root@vps-0e3e8454:~# tail -f /var/log/apache2/access.log
35.91.81.156 - - [23/Jan/2024:04:56:10 +0000] "GET /filedetails.php?repname=SvarDOS&path=%2Fwebsite%2Fphpamb.php&peg=424 HTTP/1.1" 404 438 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15"
3.224.220.101 - - [23/Jan/2024:04:56:10 +0000] "GET /blame.php?path=/pkg/&peg=1602&repname=SvarDOS&rev=250 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
110.249.202.63 - - [23/Jan/2024:04:56:11 +0000] "GET /blame.php?path=%2Fpackages%2Fdog.svp&peg=756&repname=SvarDOS&rev=630 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
23.22.35.162 - - [23/Jan/2024:04:56:14 +0000] "GET /blame.php?path=/pkg/&peg=1601&repname=SvarDOS&rev=271 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
110.249.202.202 - - [23/Jan/2024:04:56:15 +0000] "GET /blame.php?path=%2Fpackages%2Fcore%2Flabel.svp&peg=887&repname=SvarDOS&rev=791 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
3.224.220.101 - - [23/Jan/2024:04:56:19 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=301 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
35.91.81.156 - - [23/Jan/2024:04:56:20 +0000] "GET /rss.php?repname=SvarDOS&path=%2Ffiles%2Fusb-zip.mbr&peg=424 HTTP/1.1" 404 438 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15"
52.70.240.171 - - [23/Jan/2024:04:56:22 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=302 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
111.225.148.17 - - [23/Jan/2024:04:56:25 +0000] "GET /blame.php?path=%2Fpackages%2Fpkgnet-20220216.svp&peg=756&repname=SvarDOS&rev=729 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
3.224.220.101 - - [23/Jan/2024:04:56:26 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=613 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
35.91.81.156 - - [23/Jan/2024:04:56:30 +0000] "GET /rss.php?repname=SvarDOS&path=%2Fsvarcom%2F&peg=424 HTTP/1.1" 404 438 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15"
52.70.240.171 - - [23/Jan/2024:04:56:30 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=614 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
111.225.148.139 - - [23/Jan/2024:04:56:31 +0000] "GET /blame.php?path=%2Fpackages%2Ffreedoom.svp&peg=785&repname=SvarDOS HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
52.70.240.171 - - [23/Jan/2024:04:56:34 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=615 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
111.225.149.88 - - [23/Jan/2024:04:56:35 +0000] "GET /blame.php?path=%2Fpackages%2Fmsedit.svp&peg=771&repname=SvarDOS&rev=671 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
110.249.202.73 - - [23/Jan/2024:04:56:35 +0000] "GET /blame.php?path=%2Fpackages%2Fsvarcom-2022.3.svp&peg=1112&repname=SvarDOS&rev=1102 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
23.22.35.162 - - [23/Jan/2024:04:56:38 +0000] "GET /blame.php?path=/pkg/&peg=1599&repname=SvarDOS&rev=627 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
35.91.81.156 - - [23/Jan/2024:04:56:40 +0000] "GET /log.php?repname=SvarDOS&path=%2Fhelp%2Frebuild.sh&peg=424 HTTP/1.1" 404 438 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15"
110.249.201.178 - - [23/Jan/2024:04:56:41 +0000] "GET /log.php?path=%2Fwebsite%2Fmateuszbb-common.css&peg=1477&repname=SvarDOS&rev=1242 HTTP/1.1" 404 494 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
23.22.35.162 - - [23/Jan/2024:04:56:43 +0000] "GET /blame.php?path=/pkg/&peg=1600&repname=SvarDOS&rev=263 HTTP/1.1" 404 457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"
bttrx commented 10 months ago

Is there no robots.txt in place? What about kicking those bots/spiders out via .htaccess?

bttrx commented 10 months ago

$tempDir is not removed if connection is closed during processing some download. · Issue #141 · websvnphp/websvn · GitHub

Did you read this one?

mateuszviste commented 10 months ago

Is there no robots.txt in place? What about kicking those bots/spiders out via .htaccess?

I do not think that forbidding access to the resource is a solution - be it throught robots, htaccess or any other means, since the service is supposed to be available to anyone. The problem here is the software solution that breaks too easily.

$tempDir is not removed if connection is closed during processing some download. · Issue #141 · websvnphp/websvn · GitHub

Did you read this one?

I did not (until you pointed out at it), I only disabled the thing as soon as I understood who the culprit is. I'd rather avoid doing (too much) Linux administration, that's why switching to a hosted solution seemed an easy route. But if it really is just a single bug in websvn, then it is worth checking it out. Thank you for bringing it up, I will test it if solves the issue. If so, we could stay in the same configuration as we were until two days ago.

roytam1 commented 10 months ago

$tempDir is not removed if connection is closed during processing some download. · Issue #141 · websvnphp/websvn · GitHub

Did you read this one?

oh cool thanks for pointing this out. I do have my own websvn and local svn repo, but I just use robots.txt to prevent indexing websvn, and I migrated my local svn repos to git after my main PC storage crashed some years ago.

mateuszviste commented 10 months ago

So I applied the line that is mentionned in https://github.com/websvnphp/websvn/issues/141 and enabled back svn.svardos.org:

root@vps-0e3e8454:/# grep -B4 register_shut /srv/svn.svardos.org_web/dl.php 
$tempDir = tempnamWithCheck($config->getTempDir(), 'websvn');

@unlink($tempDir);
mkdir($tempDir);
register_shutdown_function(removeDirectory,$tempDir);
root@vps-0e3e8454:/#

Unfortunately, the issue still occurs, albeit slower. Within 40 minutes the size of the systemd-apache2 directory grew to 294M:

root@vps-0e3e8454:/# date ; du -ksh /tmp
Wed Jan 24 12:59:59 UTC 2024
104K    /tmp
root@vps-0e3e8454:/# 

root@vps-0e3e8454:/# date ; du -ksh /tmp
Wed Jan 24 14:04:17 UTC 2024
294M    /tmp
root@vps-0e3e8454:/#

I am not sure if the issue slowed because of the patch, or because the spiders noticed that the site is down and stopped hammering it. I still see some spider activity, but less - and notably no presence of the Amazon bot.

I will leave this on for a couple of hours to see where it leads, but so far it sadly does not look like a viable solution.

That being said, I have noticed that a new version of websvn have been released last December. Maybe it fixes the issue, so as a next step I will try to upgrade it.

mateuszviste commented 10 months ago

Unsurprisingly the tmp storage of websvn kept growing:

root@vps-0e3e8454:/srv/svn.svardos.org_websvn/websvn-2.8.3/include# du -ksh /tmp
325M    /tmp

I have upgraded websvn to the latest available version, 2.8.3. Since then (15 minutes so far) no further tmp grow. So we might be good now. I will keep an eye on this but hopes are high.

root@vps-0e3e8454:/# date ; du -ksh /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MDPdsh/tmpWed Jan 24 15:50:42 UTC 2024
56K     /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MDPdsh/tmp
root@vps-0e3e8454:/#

root@vps-0e3e8454:/# date ; du -ksh /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MDPdsh/tmp
Wed Jan 24 16:05:13 UTC 2024
56K     /tmp/systemd-private-80d02f1d944b49b6a88737622a782d66-apache2.service-MDPdsh/tmp
root@vps-0e3e8454:/#

Interestingly, the patch mentionned in https://github.com/websvnphp/websvn/issues/141 is not present in the 2.8.3 version, so I guess they solved it another way.