Backup repository check fails when server is running on OS X

hollie commented 1 year ago

Output of `rest-server --version`

Docker based install with image restic/rest-server:0.11.0

How did you run rest-server exactly?

Using Docker version 20.10.23, build 7155243 on OS X

Rest-server is started from docker compose as follows:

  rest-server:
    image: restic/rest-server:0.11.0
    volumes:
      - /Volumes/Repositories/rest-server:/data
    environment:
      - OPTIONS=--private-repos --debug
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true

What backend/server/service did you use to store the repository?

A disk volume mounted in the docker image.

Expected behavior

After running a successful backup I expect restic check to work as expected, telling me that my snapshots are 'sane' if they are and only report errors when there are actually errors in the repository.

Actual behavior

When I run a restic check on a repo that I access via rest-server the restic program reports 404 errors and missing packs although the repository is sane.

This is according to me due to the fact that OS X creates .DS_Store files in every folder on the system as part of its internal housekeeping. rest-server attempts to open those files as a folder, which fails resulting in a 404 error being reported by the restic client.

Note that I am sure that my repository is sane because if I run the check command with direct file access to the volume that is mounted in the docker image it reports no errors. So it is rest-server that is causing the errors to be reported.

The restic command will return errors like:

List(data) returned error, retrying after 552.330144ms: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 1.080381816s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 1.31013006s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 1.582392691s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 2.340488664s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 4.506218855s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 3.221479586s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 5.608623477s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 7.649837917s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 15.394871241s: List failed, server response: 404 Not Found (404)
List failed, server response: 404 Not Found (404)

At the same time the debug log of rest-server tells you:

GET /user1/data/
listBlobs()
listBlobsV2()
readdirent /data/user1/data/.DS_Store: not a directory

This error appears as many times as the restic client reports a List(data) error.

Steps to reproduce the behavior

Run restic server via the docker compose example on OS X, make a backup, wait until OS X has indexed the folders and created the .DS_Store files, run restic check.

Do you have any idea what may have caused this?

OS X is creating system-specific files that are used to store metadata and rest-server only expects folders to be present in the repository. rest-server tries to open the .DS_Store file as a folder and this fails.

Do you have an idea how to solve the issue?

Ignore potentially present OS X specific .DS_Store files when serving files to restic, or at least don't try to open a file as a folder.

Did rest-server help you today? Did it make you happy in any way?

Yes it does. First of all: this project rocks ❤️
I am migrating our backup strategy to using restic+rest-server and except for the issue I mention here all is working fine.

Once I have it completely up and running I will document on how to setup rest-server in docker combined with a Caddy proxy to cater for https and VPN access.

hollie commented 1 year ago

I can add: if I delete the .DS_Store files in the volume where the repository is stored, all is working as expected.

wojas commented 1 year ago

Thanks for reporting. I looked at the code and I can confirm that rest-server blindly assumes that everything under 'hashed' object types is a dir with actual blobs, without checking the name and if it is actually a dir.

restic / rest-server