search

restic / rest-server

Rest Server is a high performance HTTP server that implements restic's REST backend API.
BSD 2-Clause "Simplified" License
943 stars 140 forks source link

List(data) failed #296

Closed kmlucy closed 1 month ago

kmlucy commented 1 month ago

Output of rest-server --version 

$ docker images restic/rest-server:latest -q
cd6786ee5f42
$ docker exec rest rest-server --version
rest-server version rest-server 0.13.0 compiled with go1.22.5 on linux/amd64

How did you run rest-server exactly?

docker run --rm --name rest -p 8000:8000 -e PASSWORD_FILE=/config/.htpasswd -e OPTIONS='--tls --tls-cert /certs/fullchain.pem --tls-key /certs/privkey.pem' -v /etc/localtime:/etc/localtime:ro -v /mnt/storage/backup/restic:/data -v /opt/rest:/config -v /opt/dehydrated/certs/orca:/certs:ro restic/rest-server
Data directory: /data
Authentication enabled
Loaded htpasswd file /config/.htpasswd
Private repositories disabled
start server on [::]:8000
TLS enabled, private key /certs/privkey.pem, pubkey /certs/fullchain.pem

What backend/server/service did you use to store the repository?

ext4

Expected behavior

No errors

Actual behavior

I am getting list(data) errors when connecting through the rest-server, but no errors on the local machine.

Steps to reproduce the behavior

Running restic check through the rest backend was what led me to first discover this issue:

using temporary cache in /mnt/storage/tmp/cache/restic/restic-check-cache-1660270636
create exclusive lock for repository
load indexes
check all packs
List(data) returned error, retrying after 687.828349ms: List: Get "[REST]": context canceled
List(data) returned error, retrying after 1.599105373s: List: Get "[REST]": context canceled
List(data) failed: List: Get "[REST]": context canceled
List: Get "[REST]": context canceled
pack 9fad96b2d2b0c5e36e76d3e6e9bc95f7a5ee38ff587b3b5fda7848d6dc1678f0: does not exist
pack 417eeddac74cbfbb842b04b8d1946c525bc83d54e63648298849a2c099be4b63: does not exist
[REPEAT MANY THOUSAND LINES]
pack d9a08ec60d00b685159940bee6331c522d9b177096fb7d2cfc62e2baa9b7dc51: does not exist
pack 5454006973aeac3583d152f7f7c21cfd9641bb87718559c10dac4f775b076fb6: does not exist
pack a4d622046c6d27bcdb01e7fcbf8683347ca09456aac23ebec834e8fd0c8b9dee: does not exist
pack 7939abd14cde85bf710926ea7c7c79ac03d26f94cd9fc782637a6c6146d17e39: does not exist
pack 3ccea324c865a6bd6755157084952edc101ae2d1c7f82d0a369de3f1cfc10fdc: does not exist
pack b5b6959cc1373aff51889d2e76e4e0fb6dda670fff6ec6e9b634c0ebcfd7016e: does not exist
pack 4d0160052d10527f05ec4473415fd80b0ea9718003fac0b85f20f1a6f33561cd: does not exist
pack bf1716a2c1a3804260e75fc178644cc254d7a74b1c8b1b44122b479d2265b8b9: does not exist
pack 315dda8b42885d458dc92f3bda6bad7dd603d0e65ca98cd3535081f9dd04dc90: does not exist
check snapshots, trees and blobs
[38:17] 100.00%  11 / 11 snapshots

The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .

Fatal: repository contains errors

Running restic find on one of the 'missing' packs over rest-server:

$ restic find --pack 95f3a150d40ea4a3415b713409711aad6bc21395faa2cd653769a64d5205a4db
repository 6d3985ed opened (version 2, compression level auto)
[1:30] 100.00%  372 / 372 index files loaded
List(data) returned error, retrying after 524.900512ms: List: Get "[REST]": context canceled
List(data) returned error, retrying after 2.887577512s: List: Get "[REST]": context canceled
List(data) failed: List: Get "[REST]": context canceled
Get "[REST]": context canceled
List
github.com/restic/restic/internal/backend/rest.(*Backend).List
        /restic/internal/backend/rest/rest.go:324
github.com/restic/restic/internal/backend/logger.(*Backend).List
        /restic/internal/backend/logger/log.go:60
github.com/restic/restic/internal/backend/retry.(*Backend).List.func1
        /restic/internal/backend/retry/backend_retry.go:258
github.com/restic/restic/internal/backend/retry.(*Backend).retry.func1
        /restic/internal/backend/retry/backend_retry.go:132
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess.func1
        /restic/internal/backend/retry/backend_retry.go:53
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:18
github.com/cenkalti/backoff/v4.doRetryNotify[...]
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:88
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:61
github.com/cenkalti/backoff/v4.RetryNotify
        /home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:49
github.com/restic/restic/internal/backend/retry.retryNotifyErrorWithSuccess
        /restic/internal/backend/retry/backend_retry.go:62
github.com/restic/restic/internal/backend/retry.(*Backend).retry
        /restic/internal/backend/retry/backend_retry.go:130
github.com/restic/restic/internal/backend/retry.(*Backend).List
        /restic/internal/backend/retry/backend_retry.go:257
github.com/restic/restic/internal/repository.(*Repository).List
        /restic/internal/repository/repository.go:830
main.(*Finder).packsToBlobs
        /restic/cmd/restic/cmd_find.go:412
main.runFind
        /restic/cmd/restic/cmd_find.go:608
main.init.func12
        /restic/cmd/restic/cmd_find.go:43
github.com/spf13/cobra.(*Command).execute
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main
        /restic/cmd/restic/main.go:125
runtime.main
        /usr/local/go/src/runtime/proc.go:271
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

And the debug logs from rest-server while running the previous command:

HEAD /pool/config
checkConfig()
GET /pool/keys/
listBlobs()
listBlobsV2()
GET /pool/keys/ecdec7eca8457b342abc29d41b07c61d544598908085b70ccdfd228587b581b9
getBlob()
GET /pool/config
getConfig()
GET /pool/locks/
listBlobs()
listBlobsV2()
GET /pool/locks/44ec1a0b71b2d205b5a674b066f88027907518f66097eee33f60dfc1a9c8f762
getBlob()
POST /pool/locks/1fb5cc8c4cbf12d9a3012d6778e902255b72441382e7639c95dbe845222ca352
saveBlob()
GET /pool/locks/
listBlobs()
listBlobsV2()
GET /pool/locks/44ec1a0b71b2d205b5a674b066f88027907518f66097eee33f60dfc1a9c8f762
getBlob()
GET /pool/snapshots/
listBlobs()
listBlobsV2()
GET /pool/index/
listBlobs()
listBlobsV2()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/448a5b7b9742cfd64a9d77dcdbf838866675d83a78b64cec6c83abf413b481e6
saveBlob()
DELETE /pool/locks/1fb5cc8c4cbf12d9a3012d6778e902255b72441382e7639c95dbe845222ca352
deleteBlob()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/73bb041a8f900064568bc958a9fe7a3f1d6ddc39349ec80411697dd4f41ecd1a
saveBlob()
DELETE /pool/locks/448a5b7b9742cfd64a9d77dcdbf838866675d83a78b64cec6c83abf413b481e6
deleteBlob()
GET /pool/data/
listBlobs()
listBlobsV2()
POST /pool/locks/d64dcc8a9f90480bcafc7a12cc407bb15de6c61484831123c556de01c591c4da
saveBlob()
DELETE /pool/locks/73bb041a8f900064568bc958a9fe7a3f1d6ddc39349ec80411697dd4f41ecd1a
deleteBlob()
DELETE /pool/locks/d64dcc8a9f90480bcafc7a12cc407bb15de6c61484831123c556de01c591c4da
deleteBlob()

I've spot checked the 'missing' packs in the data directory, and they all seem to be there. Running check locally on the server seems to confirm this as it reports no errors.

Do you have any idea what may have caused this?

Possibly coincides with the 0.13 release. [Edit] I reverted to 0.12.1 with the same results, so that is not the issue.

Do you have an idea how to solve the issue?

No

Did rest-server help you today? Did it make you happy in any way?

This has been a perfect solution to my backups, and I've switch almost exclusively to restic with a combination of B2 and rest-server backends across Windows, Android, and Linux.

MichaelEischer commented 1 month ago

Which restic version are you using? That rather looks like a bug in restic. Please open an issue there.

kmlucy commented 1 month ago

I was on v0.17, but you are correct, reverting to v0.16.5 resolves the issue. I'll open an issue over there.

MichaelEischer commented 1 month ago

Closing in favor of the restic issue, even though a proper fix will likely also require changes to rest-server.