UploadPartCopy S3 Operation Support #286

Open axelrindle opened 1 year ago

Background

What is the problem/pain point?

I'm currently running a private GitLab instance with Storj DCS as it's object storage backend. While this is generally working, the multi-threaded upload feature is not usable due to the lack of the UploadPartCopy operation.

Having this feature enabled within Gitlab produces errors similar to the following:

WARNING: Uploading artifacts as "archive" to coordinator... POST [redacted]: 400 Bad Request (Expected(200) <=> Actual(501 Not Implemented)
excon.error.response
  :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Key>[redacted]</Key><BucketName>[redacted]</BucketName><Resource>[redacted]</Resource><RequestId>[redacted]</RequestId><HostId></HostId></Error>"
  :cookies       => [
  ]
  :headers       => {
    "Accept-Ranges"           => "bytes"
    "Content-Length"          => "524"
    "Content-Security-Policy" => "block-all-mixed-content"
    "Content-Type"            => "application/xml"
    "Date"                    => "Sun, 18 Dec 2022 00:32:17 GMT"
    "Server"                  => "MinIO"
    "Vary"                    => "Origin"
    "X-Amz-Request-Id"        => "[redacted]"
    "X-Xss-Protection"        => "1; mode=block"
  }
  :host          => "gateway.storjshare.io"
  :local_address => "[redacted]"
  :local_port    => 46860
  :path          => "[redacted]"
  :port          => 443
  :reason_phrase => "Not Implemented"
  :remote_ip     => "185.244.226.3"
  :status        => 501
  :status_line   => "HTTP/1.1 501 Not Implemented\r\n"
)  id=1205 responseStatus=400 Bad Request status=400 token=[redacted]
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument

I have requested to add a note to the official Gitlab documentation about this.

Who is impacted?

Gitlab Administrators

What is the impact?

Supporting the UploadPartCopy operation would be another step towards Storj being a true drop-in replacement for S3.

Why now?

Gitlab can be easily configured not to use the UploadPartCopy operation which makes this a low priority issue.

Nevertheless it would be nice to have this feature enabled for performance improvements.

Requirements

User Story

As a Gitlab Administrator, I want to be able to use the multi-threaded upload feature Gitlab provides in combination with Storj DCS.

Acceptance Criteria

Uploads from Gitlab to Storj DCS via the S3 Gateway work with the multi-threaded upload feature enabled.

Success Metrics

Hi @axelrindle! Thanks for this report. We are currently tracking progress on implementing support for UploadPartyCopy here. We will update the compatibility table to reflect that this is currently planned/in progress.

@amwolff That's awesome! I will add a note on that to the Gitlab documentation.

See storj/roadmap#40

@amwolff Would you mind giving an update on when this is estimated to reach production?

@axelrindle I don't have an estimate, unfortunately. As soon as we progress on this, it will be reflected on the roadmap. Internally we came up with a blueprint and an action plan, but implementing it hasn't been prioritized yet.

Would you be interested in having this working even if it's not optimal? One thing we can do is to make it work on the gateway side only, but this means that the gateway would download and upload the part being copied (and you would have to use gateway-st as we cannot enable that at the hosted gateway).

@amwolff Thank you for the update. The fact that you already have a plan internally is good enough for me :smile:

Would you be interested in having this working even if it's not optimal?

I would actually be interested in that, but I'm afraid I don't have the resources to selfhost a gateway-st instance.

If you're interested in seeing this running in a shorter timeline, can you tweak GitLab to avoid multipart moves (UploadPartCopy)? Perhaps this is as simple as a one line tweak?

https://github.com/carrierwaveuploader/carrierwave-aws/blob/bf91ca8784a8cd8c6718c603ebe0fe60317ffe75/lib/carrierwave/storage/aws_options.rb#L29

[From my quick check, it seemed like GitLab uses this library.]

@wthorp While GitLab itself can be configured to not use the UploadPartCopy operation (which is working fine), the underlying Docker Registry can not.

I'm currently running into errors similar to the ones described in distribution/distribution#1948 as well as distribution/distribution#3200. At this point I'm not entirely sure whether the registry errors are related to this specific issue...

This comment seems interesting.

Also I'm receiving the following errors now:

2023-03-20_21:13:06.51361 time="2023-03-20T21:13:06.513Z" level=info msg="S3: retrying after error" delay_s=0.284900606 error="NotImplemented: A header you provided implies functionality that is not implemented\n\tstatus code: 501, request id: [redacted], host id: "

storj / edge