Could not download file using ceph

[bluven]

Describe the bug After uploading file, I tried to download file, tusd reports error like this:

NoSuchKey: 
    status code: 404, request id: tx000000000000000022c0f-00605304fa-296a6-my-store, host id: 

To Reproduce Steps to reproduce the behavior:

1. tusc client http://127.0.0.1:18080/files kibana-7.6.2-linux-x86_64.tar.gz -r --chunk-size=5242880
2. Use browser to download file using the link returned by tusc

Expected behavior File should be downloaded.

Setup details Please provide following details, if applicable to your situation:

• Operating System: Ubuntu

• Used tusd version: I built myself

• Used tusd data storage: Ceph

• Used tusd configuration:

  export AWS_ACCESS_KEY_ID=X9OGMOAFERCT6Y09GYV0
  export AWS_SECRET_ACCESS_KEY=ALg6uCkPdM8PbqVlVsIuFN23ENCeJ2Ta1woVtW0F
  export AWS_REGION=us-east-1
  go build -o e-tusd  ./cmd/e-tusd
  ./e-tusd -s3-bucket=bluven \
  -s3-endpoint=http://10.20.8.169:32222 \
  -port=18080 \

  PS: I add an extra endpoint to implement resumable download.

• Used tus client library: tusc

---

I checked s3store's source code and found that:

        parts, err := store.listAllParts(ctx, id)
    if err != nil {
        if isAwsError(err, "NoSuchUpload") {
            info.Offset = info.Size
            return info, nil
        } else {
            return info, err
        }
    }

This snippet was found in fetchInfo method which called listAllParts to get parts info, since upload was finished, an error was returned. But here fetchInfo just checked NoSuchUpload error, while Ceph for some reason return an NoSuchKey error and this caused download failure.

Here is what I got from wireshark for listParts request:

Minio(yeah, I also tried on minio)

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchUpload</Code><Message>The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.</Message><Key>997d69dcbc198436b89510b97661ae82</Key><BucketName>test</BucketName><Resource>/test/997d69dcbc198436b89510b97661ae82</Resource><RequestId>166D5F4A1A38F015</RequestId><HostId>99b28da4-d263-4aed-bca3-fe265bdde85b</HostId></Error>

Ceph:
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>bluven</BucketName><RequestId>tx000000000000000022bb5-00605302c7-296a6-my-store</RequestId><HostId>296a6-my-store-my-store</HostId></Error>

Hope this will help you guys.

[Acconut]

Thanks for the detailed report. You already found the source of the bug on your own :)

However, I do not see this as a problem in tusd but a bug in Ceph. The AWS S3 API (and apparently also Minio) return the correct NoSuchUpload error when referencing a non-existent multipart upload in a ListParts call. This should be fixed in Ceph, since we are not very keen on handling implementation-specific inconsistencies in tusd.

I found some reports in Ceph's bug tracker, where they corrected the error code for multipart uploads (e.g. https://github.com/ceph/ceph/pull/32771), so I think they would be open for fixing it on their end. Please contact the Ceph's developers and bring this issue to their attention.

[bluven]

Ok! Thanks