OSError: [Errno 28] No space left on device

[JamesDBartlett3]

Hardware

• Raspberry Pi 4 Model B w/ 8GiB of RAM
• ZFS hard drive array w/ 15.9 TiB of free space

Software

• Raspberry Pi OS (Debian 11 "Bullseye")
• Linux kernel version 6.1.21-v8+
• OpenMediaVault version 6.9.2-1 (Shaitan)
• bandcampsync version 0.3.3 (installed from pip)

Error message

2023-10-06 09:41:40,514 sync [INFO] Decompressing downloaded zip "/tmp/tmpaha9l3q0" to "/tmp/tmptw08xjje" Traceback (most recent call last): File "/home/pi/.local/bin/bandcampsync", line 41, in do_sync(cookies_path, cookies, dir_path, media_format) File "/home/pi/.local/lib/python3.9/site-packages/bandcampsync/init.py", line 54, in do_sync unzip_file(temp_file.name, temp_dir) File "/home/pi/.local/lib/python3.9/site-packages/bandcampsync/download.py", line 82, in unzip_file z.extractall(decompress_to) File "/usr/lib/python3.9/zipfile.py", line 1633, in extractall self._extract_member(zipinfo, path, pwd) File "/usr/lib/python3.9/zipfile.py", line 1688, in _extract_member shutil.copyfileobj(source, target) File "/usr/lib/python3.9/shutil.py", line 208, in copyfileobj fdst_write(buf) OSError: [Errno 28] No space left on device

Diagnosis

I believe this error is due to an unfortunate collision between bandcampsync's default temp storage location and OpenMediaVault's overabundance of caution with regards to flash-based storage devices. Here's what I think is happening:

• When bandcampsync downloads a zip file from bandcamp.com, it saves that file inside a new directory in /tmp. Then, it extracts the zip file, putting the contents in another new directory in /tmp. Then, it moves the extracted files to the destination specified by the user with the -d parameter. And finally, it cleans up after itself by recursively deleting the two new directories it created in /tmp.
• When OpenMediaVault detects that it is running on a Raspberry Pi booted from a microSD card, it tries to prevent that microSD card from burning out prematurely due to excessive write cycles by creating a special RAMDisk from available system RAM, and symlinking all of the usual "chatty" filesystem locations (/tmp, /var/log, /var/spool, etc.) to that. Then, it creates a cronjob which synchronizes the contents of the RAMDisk to permanent storage on an hourly schedule.
• My Raspberry Pi 4 Model B has 8 GiB of RAM, roughly 30-50% of which is occupied by the OS and various background processes at any given time, and the remainder can be used by OpenMediaVault's RAMDisk. When bandcampsync downloads a particularly hefty file from bandcamp.com (in this case, a 2.5 GiB zip file, which unzips to a folder roughly the same size), stores it in /tmp (RAMDisk), and then also extracts that same file to /tmp, it runs out of "storage" space, regardless of how much free space is actually available in the destination directory.

Possible solution

One way to avoid this problem would be to add an optional temp directory preference parameter (-t/--temp, perhaps?) to bandcampsync, with several options:

• Put temp files in /tmp (default behavior if user does not pass -t parameter with a valid value)
• -t --in-place: Put temp files in a temp directory under the user's specified destination directory
• -t '~/myTemp': Put temp files in a custom temp directory path

Willing to help develop & test

I'm more than willing to help develop and test a solution to this problem, but I'm not sure where to start. I've done a fair amount of Python development, though I have never created a pip package or Docker container, written CI/CD tests of any kind, etc. If you can guide me through the process of getting a local dev environment up and running, then I'd be happy to take a crack at adding the -t parameter myself, and submitting a PR once it's done.

[badboy]

Have you tried setting one of TMPDIR, TEMP or TMP as an environment variable? Looks like bandcampsync uses Python's tempfile, which should respect any of those variables.

[JamesDBartlett3]

Have you tried setting one of TMPDIR, TEMP or TMP as an environment variable? Looks like bandcampsync uses Python's tempfile, which should respect any of those variables.

Yep, I already applied that workaround on my system. However, I still think every app should let users specify where it should save its files, even the temporary ones.

[meeb]

Thanks, this was a reasonable suggestion. While setting the TEMP ENV var is widely known, adding a flag to manually specify the temp directory path is a nice to have feature. This is in the newly released v0.3.4.

[JamesDBartlett3]

Wow, that was an incredibly fast turnaround, and thanks for taking my suggestion! Also, FYI, here's where I first heard about your tool the other day, just in case you'd like to engage in the conversation there: @oleksiy@sonomu.club on Mastodon

[meeb]

No problem, adding a CLI arg and a single parameter isn't exactly much work :)