search

BobbyWibowo / lolisafe

Blazing fast file uploader and awesome bunker written in node! 🚀
MIT License
317 stars 56 forks source link

[FEATURE REQUEST] S3 support #93

Closed JSH32 closed 2 years ago

JSH32 commented 4 years ago

Describe the solution you'd like Allow uploading to s3 type storage buckets. My VPS is very limited space, but I have a wasabi bucket setup and I want to utilize my space on there.

camjac251 commented 4 years ago

Is this still planned

BobbyWibowo commented 4 years ago

Never was, unfortunately. For the sole reason of me not using S3 personally. I immediately added the "help wanted" label to make it obvious that anybody else may take up the request if they want to.

camjac251 commented 4 years ago

Ah ok, my mistake. Thank you

JSH32 commented 4 years ago

Might add myself in the future if I can figure out the code

camjac251 commented 4 years ago

If you do, could it support file streams? Many other S3 implementations in PHP file uploaders upload and download files locally before its sent off to S3. With streaming though, it would speed up processing by a lot https://www.npmjs.com/package/@vestwell/s3-streams

JSH32 commented 4 years ago

I'll try to figure it out, the big issue I have with streams is that the user can cut an upload and there will be a leftover file in the S3.

camjac251 commented 4 years ago

Could that be solved with a check on cancelled upload to remove leftover files? Or possibly to upload with a .filepart extension and rename on completion? Then a simple purging of .filepart files in the admin panel?

JSH32 commented 4 years ago

OK so what I can do is parse the upload as a buffer in memory, once the upload is complete it will send the buffer to S3 as the body. Basically the same thing as what you described PHP uploaders do but it doesn't use disk space or IO

camjac251 commented 4 years ago

That's something I don't like about the PHP uploaders. For large files, you'd need a lot of ram to support it. I'm trying to only use 1-2GB of ram but be able to upload 5GB files

JSH32 commented 4 years ago

I mean you could technically upload a 5GB file with only 1GB of ram, for operations like these most of the time the entire file or most of it just gets dumped in disk/swap.

ghost commented 4 years ago

The problem with uploading and renaming with something like .filepart is many S3 providers charge per API request/change. Renaming them would cause a charge for upload, storage, and rename instead of just the upload and storage.

camjac251 commented 4 years ago

That is an error on my part. My understanding at the time wasn't that great of the different uploading methods that S3 had. I think the best now would be a multipart file upload through the API with the official SDK. Multipart ensures that the files are maxing out the port speeds and not relying on a single thread which would be slower on gigabit connections. https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property https://aws.amazon.com/sdk-for-node-js/

JSH32 commented 4 years ago

The way S3 works is that you can send a buffer as body and a key as the name along with a detected mimetype for the file. In one request it names the file and uploads it. No file is ever actually created on the filesystem and nothing is renamed.

JSH32 commented 4 years ago

By file system i mean the lolisafe host and a buffer is basically just a bunch of bytes which make up a file in the end. Its the actual way your system sees a file when it is read/opened and its how SDK would upload a physical file anyway.

camjac251 commented 3 years ago

I found this today. Thanks to @metal6 it is incorporated in another fork. Would it be possible to add it as an option to this fork as well? Their repo is https://github.com/weebsquad/msafe The S3 config is referenced in the config.sample.js https://github.com/weebsquad/msafe/blob/master/config.sample.js#L36-L51 and the filesystem route https://github.com/weebsquad/msafe/blob/master/routes/s3.js

metal0 commented 3 years ago

I found this today. Thanks to @metal6 it is incorporated in another fork. Would it be possible to add it as an option to this fork as well? Their repo is https://github.com/weebsquad/msafe The S3 config is referenced in the config.sample.js https://github.com/weebsquad/msafe/blob/master/config.sample.js#L36-L51 and the filesystem route https://github.com/weebsquad/msafe/blob/master/routes/s3.js

I'm not familiar with this fork of lolisafe but in my oppinion you shouldn't base any s3 code handling off my codebase, there's many issues you would need to solve in this case (the main one being the wrapper I use doesn't function at all)