search

giampaolo / pyftpdlib

Extremely fast and scalable Python FTP server library
MIT License
1.68k stars 263 forks source link

Python blocks all disk access when no space left on device #619

Closed rvdrijst closed 4 months ago

rvdrijst commented 10 months ago

We're running into issues when the disk runs out of space while uploading.

The disk we're using is an xfs network volume that is mounted to a particular directory that the ftp server has access to.

Everything works fine until the disk is full: all access to the disk is frozen. df -h hangs or shows space that is not actually available (after reboot it shows 0). Normal reboot doesn't work, we have to restart the VPS through the management console, and run xfs_repair on the disk to get it working again.

This was not an issue with the 'normal' FTP server. We added pyftpdlib to add some custom authentication and everything is working very well, until the disk is full (happens a few times a month).

It seems that python is somehow not releasing access to the filesystem when this happens, but I don't see why. Any help would be greatly appreciated. If there are other logs that I can upload that would help, let me know.

In the console of the VPS (before rebooting) Screenshot 2024-01-24 at 15 17 30

Logs of the ftp service: Screenshot 2024-01-24 at 15 23 55

giampaolo commented 4 months ago

What do you mean exactly by "blocks disk access"? That the FTP server gets stuck and does nothing?

Note: pyftpdlib is a non-blocking server. It doesn't play nice with NFS / network directories. It will block. So you probably want to use the thread-FTP server variant instead: https://pyftpdlib.readthedocs.io/en/latest/tutorial.html#changing-the-concurrency-model

rvdrijst commented 4 months ago

I don't know what happens exactly, but the python process that runs into the 'disk full' problem never releases the file system. Because the file system is locked, we cannot delete files to make room. Rebooting doesn't work either because it's waiting for the filesystem to be released. Killing the python process also doesn't help. It's properly stuck.

We work around it for now by checking the disk space before allowing upload, and simply deny the upload if there is not a certain amount of space left (and send a notification to the admin) This allows us to make room before the problem occurs.

I'll look into the thread-FTP server as well, thanks for the suggestion!

giampaolo commented 4 months ago

Rebooting doesn't work either because it's waiting for the filesystem to be released.

Wow. This sounds like a much bigger problem then. Closing this out since it doesn't seem related to pyftpdlib per se.