gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.79k stars 232 forks source link

Alias blobstore breaks multipart uploads to S3 #655

Open jrtcppv opened 5 months ago

jrtcppv commented 5 months ago

Using the following s3proxy.conf for an AWS S3 bucket:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2

I am able to complete a multipart upload successfully with aws s3 cp ~/big_file.mp4 s3://test-bucket --endpoint-url http://localhost:8085

However, when I enable alias-blobstore and bucket-locator:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2
s3proxy.alias-blobstore.project-1=test-bucket
s3proxy.bucket-locator.1=project-1

And attempt to upload with aws s3 cp ~/big_file.mp4 s3://project-1 --endpoint-url http://localhost:8085 I receive the following error after uploading seemingly all parts:

upload failed: ../../big_file.mp4 to s3://project-1/big_file.mp4 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation: The specified key does not exist.

Not sure if this is relevant but I am invoking s3proxy using the docker container with the latest tag, like so:

docker run --rm -v ~/s3proxy.conf:/s3proxy.conf --publish 8085:8085 --entrypoint /bin/bash andrewgaul/s3proxy -c "java -jar /opt/s3proxy/s3proxy --properties /s3proxy.conf"
jrtcppv commented 5 months ago

I just tried with the bucket locator removed, so just using the alias-blobstore is sufficient to reproduce:

s3proxy.endpoint=http://0.0.0.0:8085
s3proxy.authorization=none
jclouds.provider=aws-s3
jclouds.identity=####################
jclouds.credential=####################################
jclouds.region=us-east-2
s3proxy.alias-blobstore.project-1=test-bucket
gaul commented 3 months ago

@timuralp could you look into this?