search

restic / rest-server

Rest Server is a high performance HTTP server that implements restic's REST backend API.
BSD 2-Clause "Simplified" License
978 stars 143 forks source link

Bad request reply on disk full #155

Closed Enrico204 closed 3 years ago

Enrico204 commented 3 years ago

Output of rest-server --version

rest-server 0.10.0 compiled with go1.15.2 on linux/amd64

How did you run rest-server exactly?

$ ./rest-server --append-only --listen :8443 --no-auth --tls --tls-cert cert.pem --tls-key key.pem --path /tank/restic/
Data directory: /tank/restic/
Authentication disabled
Private repositories disabled
TLS enabled
Private key: key.pem
Public key(certificate): cert.pem
Starting server on :8443

When executed with --debug, it says also (see below):

GET /index/
ListBlobs()
ListBlobsV2()
GET /snapshots/
ListBlobs()
ListBlobsV2()
POST /data/8dda82c5094805b46ae06f91be87fa74c0514abb0d3ef90d0e111d0c4b2714e7
SaveBlob()
write /tank/restic/data/8d/8dda82c5094805b46ae06f91be87fa74c0514abb0d3ef90d0e111d0c4b2714e7: no space left on device
DELETE /data/8dda82c5094805b46ae06f91be87fa74c0514abb0d3ef90d0e111d0c4b2714e7
DeleteBlob()
POST /data/8dda82c5094805b46ae06f91be87fa74c0514abb0d3ef90d0e111d0c4b2714e7
SaveBlob()
write /tank/restic/data/8d/8dda82c5094805b46ae06f91be87fa74c0514abb0d3ef90d0e111d0c4b2714e7: no space left on device

What backend/server/service did you use to store the repository?

ZFS dataset

Expected behavior

On disk full, rest-server should reply something like HTTP status 5xx

Actual behavior

Currently, rest-server replies 400 Bad request

Steps to reproduce the behavior

Create a partition / ZFS dataset / any store with low capacity. Backup a very big set of data. When the backend partition is full, restic client is saying:

Save(<data/8dda82c509>) returned error, retrying after 552.330144ms: server response unexpected: 400 Bad Request (400)

In my case, the data I'm saving is 800 GB, the backend partition size is 750 GB.

Do you have any idea what may have caused this?

The backend storage capacity is full, however the error is mishandled (4xx is a client error, this is actually a "server" error)

Do you have an idea how to solve the issue?

rest-server should reply with 5xx as "disk full" is a "server error" (it can't be solved by the client itself).

Did rest-server help you today? Did it make you happy in any way?

Yes, I'm using it day-by-day to backup all my RPis and my laptop at home at full speed :-D

Enrico204 commented 3 years ago

If you think that the solution is correct (changing 400 Bad request to a 5xx error), I can prepare a pull request.

Also, I suggest to use 507 Insufficient Storage error (RFC 4918, WebDAV). Even if we are not using WebDAV, we can use this code as it's standard, and it's clear enough to clients :-)

Enrico204 commented 3 years ago

Note: I saw the current work on quota manager on master branch, and I think that the error code when out of quota should be switched to 507 Insufficient storage.

Currently, "413 Request entity is too large" (the currently used error code) seems to imply that the HTTP request is too large, which is not the case. I think that "Insufficient storage" will be more appropriate in this case.

I can include this in the pull request as well, if you agree :-)

wojas commented 3 years ago

507 Insufficient Storage would definitely be more helpful to a user.

This should also be applied to Write errors that return ENOSPC or EDQUOT and explain which of the errors caused it in the end, but this will require some OS specific syscall checks.

Enrico204 commented 3 years ago

I created a pull request with a proposal for the change :-)

Feel free to add comments! I haven't written any test or documentation yet, I'll do it :-)

Enrico204 commented 3 years ago

The pull request was merged, this change has been implemented :tada: