Closed Krumbelfix closed 1 year ago
I just found the solution: The Mount was full.
after increasing the storage capacity i can prune the backup:
on the backupserver:
lvextend --resizefs -L 310G /dev/backup/lv-docker-backup
restic -r sftp:root@<snipped-ip-adress>:/mnt/backup-docker/ --verbose prune --password-file=/root/.restic/backup-docker.pwd
As an improvement: an warning or hint when the storage is (nearly) full would be very useful.
Hmm, the generic error message really doesn't help. I thought there has been an older issue with the same problem, but couldn't find it. To provide any useful error message we'd need to find a way to detect what the problem is, "failure" is way to generic.
The problem is in OpenSSH's sftp-server program, which sends a generic failure code for ENOSPC with no way to retrieve the cause, even though the protocol has a code reserved for this case. restic could do a StatVFS before writing a file, as a heuristic.
Sorry if I understood wrong how this was closed. I experienced the same error msg today, by googling I got here and indeed the mount was full. Better error msg would be helpful :)
@ajsicek restic 0.12.1 got a best-effort fix for this issue. If you still get it from a newer restic version, please report that version and information about the SSH implementation on your server. Some just send very generic error codes that restic can't figure out.
hi, not sure exactly what info is relevant both are debian stable servers most of the software is the same Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux restic 0.14.0 compiled with go1.19 on linux/amd64
if I run the connection over which I tried the backup eith ssh -v I get debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Debian-5+deb11u1
That's a recent enough OpenSSH. When you sftp to the server, what do df and df -i say? For reference, this is what restic checks before generating a "no free space left on device" message:
That's "no free inodes or block size * free blocks < size of file we're uploading". It might be that you're hitting ENOSPC earlier than that. In particular, some of the space on the server may be reserved for the root user.
Also, try running restic in debug mode and see if a message starting "sftp: StatVFS returned" occurs in the log.
That's "no free inodes or block size * free blocks < size of file we're uploading". It might be that you're hitting ENOSPC earlier than that. In particular, some of the space on the server may be reserved for the root user.
Isn't there usually a small reserved amount of storage which is only writable by the root user? That reserved storage would probably also be reported as free space?
to be able to do the backups. I resized the disk, so not sure, how much I can investigate any further :(
df on server now says
/dev/sdb1 131364796 68578612 56080404 56% /mnt/data_alfa
df -i says
/dev/sdb1 8364032 5164 8358868 1% /mnt/data_alfa
when the backup failed df
showed "use 100%" df -i
I did not check
in auth.log on the server there is msg from that backup time
sftp-server 1715951 - - error: process_write: write: No space left on device
anything else I can dig out of logs?
Isn't there usually a small reserved amount of storage which is only writable by the root user? That reserved storage would probably also be reported as free space?
Yes, and I just reproduced the behavior by backing up to an 8MiB image with 50% reserved blocks. sftp.StatVFS.FreeSpace counts the total free space, but the struct has the required information about reserved blocks. I suppose we can assume the user is not root on the server?
I suppose we can assume the user is not root on the server?
yes that is correct
@ajsicek I meant in general, I would not address you as "the user" :) #4075 should fix this.
Output of
restic version
restic 0.12.0 compiled with go1.15.8 on linux/amd64
How did you run restic exactly?
restic -r sftp:root@<ip-adress-snipped>:/mnt/backup-docker/ --verbose prune --password-file=/root/.restic/backup-docker.pwd
What backend/server/service did you use to store the repository?
Scaleway root-server via SFTP
Expected behavior
cleaning up / prune snapshots so the backup is working again
Actual behavior
restic -r sftp:root@<ip-adress-snipped>:/mnt/backup-docker/ --verbose check --password-file=/root/.restic/backup-docker.pwd
restic -r sftp:root@<ip-adress-snipped>:/mnt/backup-docker/ --verbose prune --password-file=/root/.restic/backup-docker.pwd
restic -r sftp:root@<ip-address-snipped>:/mnt/backup-docker/ --verbose backup /var/lib/docker/ --password-file=/root/.restic/backup-docker.pwd
Steps to reproduce the behavior
unfortunately i can't provide any useful information :(
Do you have any idea what may have caused this?
no :(
Do you have an idea how to solve the issue?
and an aditional "no" :)
Did restic help you today? Did it make you happy in any way?
I really, really love restic and its design principles. Its an great software which helps me a lot. <3