Closed timkatje closed 5 years ago
Additional information:
Since I'm now actually home and doing this in person, I was able to do some physical interventions. So I fully ejected the USB drive from within my QNAP GUI and then rebooted the entire server. Even after a full reboot, and remounting the drive, borg reports the exact same issue when I try to run create again to continue the backup where it left off:
[/share/external/DEV3401_1] # borg create --stats --progress --show-rc --compression auto,lzma,6 media::InitialBackup-2018-07-26 /share/CACHEDEV1_DATA/media/
Enter passphrase for key /share/external/DEV3401_1/media:
Local Exceptionche transaction: Reading chunks
Traceback (most recent call last):
File "borg/archiver.py", line 4312, in main
File "borg/archiver.py", line 4244, in run
File "borg/archiver.py", line 152, in wrapper
File "borg/archiver.py", line 547, in do_create
File "borg/archiver.py", line 511, in create_inner
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 598, in _process
File "borg/archive.py", line 1028, in process_file
File "borg/archive.py", line 956, in chunk_file
File "borg/archive.py", line 944, in chunk_processor
File "borg/cache.py", line 891, in add_chunk
File "borg/cache.py", line 550, in begin_txn
File "shutil.py", line 241, in copy
File "shutil.py", line 122, in copyfile
File "shutil.py", line 82, in copyfileobj
OSError: [Errno 28] No space left on device
Platform: Linux MemoryAlpha 4.2.8 #1 SMP Tue Jul 10 07:27:26 CST 2018 x86_64
Linux:
Borg: 1.1.6 Python: CPython 3.5.5
PID: 27229 CWD: /share/external/DEV3401_1
sys.argv: ['borg', 'create', '--stats', '--progress', '--show-rc', '--compression', 'auto,lzma,6', 'media::InitialBackup-2018-07-26', '/share/CACHEDEV1_DATA/media/']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
This is also why I've gotten into the habit of just reformatting the entire drive to start over fresh.
Maybe this is another case of a slightly misleading ENOSPC.
I will have a detailled look later, but maybe you could just check if there are free inodes left on the filesystem when you see this kind of error. You can choose the amount of inodes when mkfs-ing the filesystem.
@timkatje ^^^
Sorry, @ThomasWaldmann, I expected to get a email notification or something telling me that you'd replied. I'll have to do some research on inodes and mkfs-ing. Since it's a consumer NAS device, most of that stuff is abstracted away from me, and I'm still pretty novice with Linux OSs.
For instance, when I first added the USB drive, I just used the QNAP web-GUI's format option and selected EXT4 as the filesystem as opposed to the NTFS that the drive arrived with (gave me a couple-hundred extra usable MB and prefer to keep it all Linux native). But that's the extent of options within the GUI, just filesystem type (EXT4, EXT3, FAT32, NTFS and HFS+), and encryption.
To guide me in the right direction, is the limited inode thing system-wide, or specific to each individual filesystem (which I'm interpreting to be each different "partition" on the attached drives). And then would it be the source, or the target filesystem that's most likely at fault for this (I'd assume target).
Would using a different filesystem on the USB drive perhaps have a similar affect to manipulating the number of inodes? Maybe I should just try EXT3 or NTFS...
As it is blowing up in the cache code, check the filesystem that has ~/.cache/borg
.
It could be either simply full (really no space free) or out of free inodes.
And in case you ran into trouble with ~/.cache/borg/<REPOID>/chunks.archive.d
eating up all your space:
You can simply rm -rf
that directory and replace it with an empty file, then borg will stop having stuff there (but it will be slower for the cases when that was used).
So, I started off by attempting to just delete the chunks.archive.d directory. I was able to use "borg create" to start another backup job, without formatting the disk, and it actually began the process. Before deleting the chunks.archive.d directory, it refused to work, as discussed above.
But, after a period of time, the "No space left on device" error popped up again. Here's the output:
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc/chunks.archive.d] # cd ..
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # rm -rf chunks.archive.d/
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # ls
chunks config files README txn.tmp/
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # borg create --progress --stats --show-rc /share/externalDEV3401)1/media::secondtrybackup /share/CACHEDEV1_DATA/medi>
-sh: syntax error near unexpected token `)'
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # borg create --progress --stats --show-rc /share/external/DEV3401)1/media::secondtrybackup /share/CACHEDEV1_DATA/med>
-sh: syntax error near unexpected token `)'
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # borg create --progress --stats --show-rc /share/external/DEV3401_1/media::secondtrybackup /share/CACHEDEV1_DATA/med>
Enter passphrase for key /share/external/DEV3401_1/media:
Local Exceptionche transaction: Reading files
Traceback (most recent call last):
File "shutil.py", line 122, in copyfile
File "shutil.py", line 82, in copyfileobj
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "borg/archiver.py", line 4312, in main
File "borg/archiver.py", line 4244, in run
File "borg/archiver.py", line 152, in wrapper
File "borg/archiver.py", line 547, in do_create
File "borg/archiver.py", line 511, in create_inner
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 621, in _process
File "borg/archiver.py", line 598, in _process
File "borg/archive.py", line 1028, in process_file
File "borg/archive.py", line 960, in chunk_file
File "borg/archive.py", line 938, in write_part_file
File "borg/archive.py", line 449, in write_checkpoint
File "borg/cache.py", line 921, in chunk_decref
File "borg/cache.py", line 552, in begin_txn
File "shutil.py", line 241, in copy
File "shutil.py", line 122, in copyfile
OSError: [Errno 28] No space left on device
Platform: Linux MemoryAlpha 4.2.8 #1 SMP Tue Jul 10 07:27:26 CST 2018 x86_64
Linux:
Borg: 1.1.6 Python: CPython 3.5.5
PID: 30501 CWD: /root/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc
sys.argv: ['borg', 'create', '--progress', '--stats', '--show-rc', '/share/external/DEV3401_1/media::secondtrybackup', '/share/CACHEDEV1_DATA/media/']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
[~/.cache/borg/a3c81b0a15dada36d0e29e653b07ab39e8880db46f374fb9b8f1ca2bd0c101fc] # ls
chunks config files README txn.tmp/
I left in some of my other filesystem operations in the transcript just in case I didn't do something right.
I have since utilized "borg delete" to remove all of my repositories from the external drive, without formatting.
I have also also set the following environment variables:
TMPDIR=/share/external/DEV3401_1/borg_tmp
BORG_CACHE_DIR=/share/external/DEV3401_1/borg_caches
I tried running borg init to create a new repository on my now mostly empty external drive, and ran into the same error as usual. Seems like the only way I'll be able to clear the error, even just temporarily, is to reformat the external drive.
I'm researching the best way to do that via SSH instead of using the QNAP OS interface.
Just read my last post again.
Hmm, looks like i misinterpreted "after a period of time" in your post. Guess you meant that you deleted stuff, ran borg create and then it happened again. I thought you meant after multiple backups (so, after hours, days, weeks).
So, did you find out the root cause meanwhile?
No response, closing.
I just had a very similar issue with my qnap nas. Running a simple borg info resulted in "No space left on device".
How to fix: Check free space with df -h
Filesystem Size Used Available Use% Mounted on
none 400.0M 400.0M 32.0k 100% /
After running rm -r ~/.cache/borg/
Filesystem Size Used Available Use% Mounted on
none 400.0M 207.9M 192.1M 52% /
Change cache dir to fix the problem https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables
You should use something in /share/CACHEDEV1_DATA as your cache dir
I have the same issue. I've removed the old cache and freed up / which is now showing 87% use. Have pointed the cache dir as:
BORG_CACHE_DIR=/share/CACHEDEV1_DATA
But that doesn't seem to work. The cache dir is still recreated in /root/.cache/borg. What am I missing?
-- BVS
EDIT: same issue here https://www.reddit.com/r/DataHoarder/comments/fxuckq/borg_backup_and_rclone_issues_with_space/
EDIT2: had a typo - can confirm the above does in fact work re-directing cache to your main storage device. For others set as:
export BORG_CACHE_DIR=/share/CACHEDEV1_DATA
Local media share backup of tv shows and movies reports OSError: [Errno 28] No space left on device on 8 TB USB drive.
Have you checked borgbackup docs, FAQ, and open Github issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
ISSUE
System information. For client/server mode post info for both machines.
Linux version 4.2.8 (root@U16BuildServer104) (gcc version 4.9.2 (toolchain config: [gcc-4.9.2 binutils-2.25 glibc-2.21])
Your borg version (borg -V).
1.1.6
Operating system (distribution) and version.
Hardware / network configuration, and filesystems used.
QNAP TS-563 w/ AMD GX-420MC SOC (amd64) and 16 GB RAM running QTS (QNAP's OS) 4.3.4.0644 Build 20180710
Backing up to Seagate Backup+ Hub 8TB via direct connect USB 3.0.
How much data is handled by borg?
~ 5TB, never make it past 600 GB though before backup errors out. Same thing has happened with smaller shares of media that I'm trying to backup, including some sub-TB shares.
Have tried smaller, sub-TB shares, encountered same thing.
Full borg commandline that lead to the problem (leave away excludes and passwords)
borg create --stats --progress --show-rc --compression auto,lzma,6 media::InitialBackup-2018-07-26 /share/CACHEDEV1_DATA/media/
borg info media
Describe the problem you're observing.
I have yet to successfully backup any of my different media shares from my QNAP NAS to a brand new 8 TB USB drive I've attached and mounted (I let QNAP mount it automatically the way that it wants to).
I've formatted the USB drive as EXT4 for best compatibility with my NAS.
At first, I was using the simplest possible commands with very few parameters, but I've since started using more features, like the auto compression.
Once this error occurs, it seems like I cannot use any borg commands, not even borg info to look at what's in the repo, just get the same error. I also don't want a bunch of stagnant data, so I usually just reformat the drive and try again.
When I check on the external device in QNAP management interface, it shows that I'm using 6.42% of my capacity after this failed backup. So it's really surprising to constantly encounter "Out of space" errors.
Once I encounter this, all subsequent borg commands, whether it be borg init to make a new repo, or borg info, reports "Out of space."
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
For me, it's as simple as running the basic commands from SSH into my QNAP NAS. Create a repo, select a source share of 500 GB+ and every time it has displayed this issue. This is the furthest I've even gotten a backup of any of my big shares to get.
I originally thought that it might've had something to do with copying my .qpkg (applications) while they were running, so I gave up on backing that share up. My ultimate goal is to just pull all of the media, of all types, from multiple shares in order to redo my RAID configuration as I've reached 80% used capacity in my current array.
Include any warning/errors/backtraces from the system logs
After checking on something in my QNAP GUI, i ran the borg info media command again and received a slightly different output. I was actually about to try manually running the break-lock command, but I guess it attempted to do so automatically this time: