Signed URL for PUT requests

[tropnikovvl]

Is your feature request related to a problem? Please describe. At the moment there is a limitation of signed URLs only for GET requests, can you tell me if there are plans to do this for PUT requests too? The fact is that I upload files larger than 100GB, sometimes 300GB each. And this is a very large load on the file system.

Describe the solution you'd like Uploading of files without overhead on the application and file system. Here's and here an approach to how you can achieve this.

[oxyno-zeta]

Hello,

Thanks for your issue. The reason I haven't made it was just: I never thought that would be useful. I can do it or you can do a merge request if you want. I cannot say when I will do it sorry.

Regards,

Oxyno-zeta

[tropnikovvl]

Hello @oxyno-zeta! Can you tell me if we can expect this feature in the foreseeable future? I noticed that in the latest release, you implemented multi-part uploading. Can you inform me whether it would be suitable for my particular use case?

I am trying to understand the impact of this new feature on traffic, disk usage, CPU and memory. Unfortunately, I haven't had the opportunity to test it yet. Thank you in advance.

[oxyno-zeta]

Hello @tropnikovvl ,

Sorry for the delay, I have only few amount of time of projects those days...

Yes this should work. S3Manager allow to upload on memory and stream upload to S3 compatible servers.

Tell me if this isn't working :)

Best regards,

[tropnikovvl]

Hello @oxyno-zeta,

I tested your new feature and everything looks great.

Unfortunately, my case is still not running, since the data is first loaded inside the s3-proxy and only then into the s3 bucket. Instead of loading them directly into the s3 bucket, immediately bypassing the s3-proxy.

In my case, this slows down data loading in general, I have to pay for additional traffic and temporary storage for the s3-proxy.

In any case, I am very grateful for your product and hope that one day you will have time to implement signed upload links!

[oxyno-zeta]

Hello @tropnikovvl ,

I'm sorry to ear that... I reflected a bit on this. I'm facing a issue:

• People will create a PUT request with form-data including a file inside a field called "file"
• They will be redirected to S3 with a pre-signed URL
• The signature isn't the same an upload will fail :(

That will imply that people will have to try an upload with a signature, catch the redirect and retry with another signature. I'm not sure that will work.

What do you think ? Am I clear ?

PS: Thanks for your return on this project !

[tropnikovvl]

@oxyno-zeta ,

I see it this way:

1. the user opens a page with a s3-proxy (no PUT request to s3, only GET)
2. the user selects a key (directory) and a file (In theory, here we still do not send a PUT request, because everything happens only in the browser)
3. The user presses the upload button. a. At this point, you should check whether the file exists in the same path, and if so, suggest overwriting or canceling the upload. b. if the file does not exist (or it is overwritten), then a presign link is created and it is also used to upload files
4. file uploaded successfully.

I may be wrong somewhere in the mechanics of the process, but judging by the AWS documentation, this should generally work

[oxyno-zeta]

Hello @tropnikovvl ,

I'm still reflecting on this subject.

The workflow you proposed is supposing that you have a clever interface and a get and upload file API that s3-proxy don't have I'm afraid... For me, this project isn't the right place for such a workflow. It hasn't been designed for this.

I'm reflecting on another possibility:

• having s3-proxy-interfaces or clever interface will be mandatory for this feature
• enable feature support in interface
• POST requests will have to be created to allow returning the url signed for the interface in order to make the upload request.

With this, API call will also be possible.

What's "problematic" for me is that:

• it's creating a new endpoint with a completely different approach
• the configuration has the notions of GET, PUT and DELETE requests but POST won't exist per se, it will be a feature under PUT
• I don't know if when this feature is enabled we should completely close the PUT endpoint or keep it open. I'm more in favor of closing it but that won't allow rolling upgrades for end users...

As you can see, I'm still searching :). If you have any comments on those lines, do not hesitate!

[tropnikovvl]

Hello @oxyno-zeta!

Unfortunately, I’m not sure that I can help you think about what to do correctly from the point of view of the project.

Your project as a whole works as required for me, but there are 2 nuances that are important to me.

• This is the ability to load data into s3 bypassing the backend (those presigned links).
• This is the ability to check that a file with the same name and path exists before loading.

I understand that this is open source and, in general, I have no right to demand or ask you to do anything. I just wanted to share my experience using the product and my opinion on how it could be improved.

Thank you!

[oxyno-zeta]

Hello @tropnikovvl ,

You can ask using issues on my project ;) . I'm very happy to see people having ideas on those projects. I'm still searching but I'm not very enthusiastic with the solution proposed before.

By the way, the ability to check that a file with the same name and path exists before loading is already possible. Check here: https://oxyno-zeta.github.io/s3-proxy/configuration/structure/#putactionconfigconfiguration with the field: allowOverride

Thank you !

[tropnikovvl]

Hello @oxyno-zeta!

Thank you very much for your responsiveness!

Perhaps you will succeed with the POST request. According to Google, this is possible.

In any case, I will be glad to hear any news!