FTP server locks up/slows down for a while after large number of downloads

[jvyden]

Whenever I make a large number of requests, e.g. downloading all param.sfo files and icon0.png files from each game, the webMAN server seems to freeze up for a while - but after waiting for a couple minutes it seems to restore itself perfectly as if nothing ever happened.

[aldostools]

Try limiting the number of concurrent connections to 2 in the FTP Client.

The FTP server in webMAN MOD is running all the time. Therefore the resources are more limited than other FTP server applications,

The recommended FTP Client is FileZilla. Although it works with other FTP clients.

[jvyden]

This is a code-controlled client from C# using FluentFTP, which I'm fairly certain is only using one connection since I only use one thread synchronously.

[jvyden]

Just to verify, webMAN's FTPd is single-threaded, correct? I only see one direct usage of ftpd_thread in main.c, but I just want to make sure there aren't any other cases I'm missing.

I've double-checked by reading the source code of the library we're using, and it looks like we're responsible for threading/managing connections (which we only create one of at a time), so it definitely seems like a bug in webMAN - especially given that we don't use async for FTP at all.

[kostirez1]

Just to verify, webMAN's FTPd is single-threaded, correct? I only see one direct usage of ftpd_thread in main.c, but I just want to make sure there aren't any other cases I'm missing.

It's actually using a single thread to accept new connections - ftpd_thread() and then one more thread per connection - handleclient_ftp(). When transferring 2 files concurrently, this implementation usually uses 4 threads total:

1. ftpd_thread() - To accept new connections on port 21 (2121, or any other, depending on settings)
2. handleclient_ftp() - The first connection client made to list files and directories - commands like MLSD, LIST,.... This connection may time out when unused for a longer period - some clients solve this by sending NOOP from time to time.
3. handleclient_ftp() - The first file transfer - commands STOR / APPE or RETR
4. handleclient_ftp() - The second file transfer, same as above

It used to be the case you could open more than 2 transfers concurrently, but that was changed due to reasons described below.

Whenever I make a large number of requests, e.g. downloading all param.sfo files and icon0.png files from each game, the webMAN server seems to freeze up for a while - but after waiting for a couple minutes it seems to restore itself perfectly as if nothing ever happened.

This is actually an intended behavior since #764 - Add FTP maximum limit for concurrent transfers. Previous versions of wMM froze the FTP server indefinitely under the same circumstances, as discussed in #760 - [Bug] FTP server disconnects and freezes. This behavior has something to do with kernel memory used to store open TCP/IP connections, and how the FTP (protocol) transfers files. FTP uses a new TCP connection for each transfer, not allowing for socket reuse.

The kernel in PS3's OS allows only ~10 connections to be open and stored/tracked at once. And for TCP, even closed connections are stored for some due to the TCP TIME_WAIT mechanism. As I understand, there's no way to remove the connection manually. Not even with SO_LINGER option set to a low value - it just ignores it.

The "waiting period" you've observed is then just the FTP server waiting for the kernel to free up some memory. 🙂

[jvyden]

Whoops, this was left open for longer than intended. Closing as non-issue. :sweat_smile:

As a side-note we just ended up resolving this by just caching the files to the disk and reading from that instead. It'll still take a while on first connection but I think that's an acceptable compromise for ease of use. Thanks for the detailed explanation on how the kernel works in this regard!