search

treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.36k stars 348 forks source link

investigate the need for UploadManager on the S3 client #986

Closed arielshaqed closed 10 months ago

arielshaqed commented 3 years ago

We are adding the optional use of UploadManager because of an issue connecting to minio[1]. Is this a Good Thing:tm:?

Ideally this flag would go away because we would find "the correct setting" for it. Alternatively we might use the upcoming AWS v2 Go client, or improve our streaming signer.

Details: We need UploadManager to put-object directly from a stream (S3.PutObject needs a io.ReadSeeker, which obviously doesn't work). So we have our own StreamSigner, which has some difficulty signing exactly the right fields with the AWS v4 module. This issue is to investigate alternatives.

[1] Possibly an issue with a proxy (nginx?) in from of minio rewriting enough of the signed headers to break the signature.

arielshaqed commented 3 years ago

Context: https://lakefs.slack.com/archives/C016726JLJW/p1606774533137000 (open Slack community)

github-actions[bot] commented 10 months ago

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

github-actions[bot] commented 10 months ago

Closing this issue because it has been stale for 7 days with no activity.