jbsparrow / CyberDropDownloader

Bulk Gallery Downloader for Cyberdrop.me and Other Sites
GNU General Public License v3.0
177 stars 14 forks source link

[BUG] no free space check when running with --retry-failed #103

Closed NTFSvolume closed 1 month ago

NTFSvolume commented 1 month ago

CDL checks free space on the drive before starting a new file download. However, when running with -retry-failed, it will download to the original download folder used when the URL was added to the DB, not the currently specified OUTPUT_FOLDER. If that folder and OUTPUT_FOLDER are on different drives, CDL will keep downloading until no space is left (0B)

Setup Information

How to reproduce (UNIX only):

  1. Create a small RAMDISK for the test, 1.5GB.
sudo mkdir /tmp/ramdisk
sudo chown $USER /tmp/ramdisk
sudo mount -t tmpfs -o size=1500m ramdisk /tmp/ramdisk
  1. Create a new config folder
mkdir -p /tmp/CDL_test_config
  1. Use CDL to download a file bigger than the RAMDISK (URLS.txt with example attached) and use the config folder created. The download will be skipped (CDL requires 5GB free by default) but it will create the default global_settings.yaml
cyberdrop-dl -i /path/to/my/URLS.txt -d /tmp/ramdisk/cdl --appdata-folder /tmp/CDL_test_config/ --download
  1. Edit global_settings.yaml and change required_free_space to 1GB
sed -i 's@required_free_space: 5@required_free_space: 1@g' /tmp/CDL_test_config/AppData/Configs/global_settings.yaml
  1. Run CDL a second time with the same parameters but cancel the download (CTRL + C) at about 600MB downloaded, when the RAMDISK has less than 1GB available. This will also add the file as incomplete (failed) to the database

  2. Run the command a third time. In this case, the download will be skipped by config cause RAMDISK does not have 1GB available anymore. That's the normal behavior

  3. Run CDL one last time but now change the OUTPUT_FOLDER to any folder on a different drive, and add the flag --retry-failed.

cyberdrop-dl -i /path/to/my/URLS.txt -d /different/download/path --appdata-folder /tmp/CDL_test_config/ --download --retry-failed
  1. Download will start as normal cause CDL checked free space on /different/download/path, not on the RAMDISK. However, the file is still downloaded to the RAMDISK.

The download will continue until the drive is completely out of space. CDL will not crash but the download will just fail with "Unknown" as error on the UI, but the logs do show "OSError: [Errno 28] No space left on device". If the database is on that drive, sometimes it gets corrupted. Probably because CDL tries to write to it at the end.

Attachments:

downloader.log URLs.txt