Open GitRon opened 6 years ago
@juliomalegria Please have a look at my PR and consider merging it. In combination with django-storages 1.6.6 and boto3 to works for Amazon S3.
@juliomalegria Any chance of getting this in the master branch and release a new version?
No update so far?
@jerinpetergeorge I fixed the merge conflicts. Whats your opinion on this topic?
I have tried the new changes in my local machine. Unfortunately, it's not uploaded to S3
requirements.txt
since I've added the relevant modules added to the demo)django-chunked-upload-demo/settings.py
and fill the following keys,
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_STORAGE_BUCKET_NAME
FileUploadTest
instance by uploading a fileCHUNKED_UPLOAD_USE_TEMP_STORAGE
settings)Did I miss something? @GitRon
@GitRon
I think we should initiate the S3 upload using upload_file()
method when we receive the last chunk from the client. But, if we try to upload the whole file to S3 at that moment, probably the NGNIX would close the connection from the browser client.
and there is another method called, upload_part()
, it seems like we could upload the chunks in realtime to the S3 bucket.
@jerinpetergeorge I'm sorry but I cant follow you. We are using boto3 as a connector and its file storage class. So you dont have to worry about the upload at all. The only thing what you need to take care about is that you cannot write the chunks to the target location.
Oops :persevere:
Have you tried to reproduce the behavior that I mentioned here ??
Anyway, let me ask you some doubt regarding this, that may help you to understand what I am trying to say
CHUNKED_UPLOAD_USE_TEMP_STORAGE
?CHUNKED_UPLOAD_USE_TEMP_STORAGE
only support boolean values, (ie True
and False
)
OR,
Could you provide a minimal example that demonstrates the S3 upload? @GitRon
Ok, I think we are talking not about the same thing here 😅
My changes are not about ENABLING cloud storag uploads. Its just prerequisite.
If you are writing to a file storage like a hard drive, it does not matter where you park the chunks. On an object storage it does because you cannot write to an object storage. It only accepts full files as an object.
I gave it a second thought and maybe we should always - without the settings flag - write the chunks to the temp storage. I think this is the main reason why there is a temp-dir, right? To contain temporary data.
What do you think?
Storing the chunks to a temporary location and moving it to the target location after completion is a good idea. I think we should implement this first before the S3 thing. Apart from that, If someone using s3 storage as their default storage system, they would expect the file to be uploaded to the S3 bucket. So we should implement a full-fledged s3 upload capability here.
Perfect, I'll change my PR then.
@jerinpetergeorge Ok, the behavior is now as follows:
If you define no storage, it takes my temporary storage. But you can override it with your settings var if you like.
I like this solution, good that we talked about it 👍
Great! but this is only half of the solution. We should move the completed file
to CHUNKED_UPLOAD_PATH
when a user uses the default settings, shouldn't we? @GitRon
@jerinpetergeorge Sound good. But doesn't it already work like this? I just collect the chunks in temp storage.
@jerinpetergeorge Somebody asked for my fix on this issue. Whats your status?
I pointed out - after thinging about the things that happended - that me might want to avoid renaming variables etc. To maybe make it stable and compatible again? What do you think? 😃
Sorry for the utter delay from my side. Thanks for your effort @GitRon and please do let me give a couple of days to recollect the project context ( :astonished: ) and review the PR as I am completely slipped away from this.
@GitRon Thanks for the PR. But, It is hard for me to review the PR since I have slipped away from the context, as I already said. Also, I don't think I can make it to work in the near future.
So, I would recommend you to ask @juliomalegria to make you a collaborator of this repo or fork this project and maintain it yourself.
@jerinpetergeorge @juliomalegria Sure, if you give me permission I'd fix the bug people were complaining about. Would you be able to deploy a new version at the time?
I won't be able to do that :disappointed: You can contact @juliomalegria via his personal mail (juliomalegria@gmail.com
got from his personal site). If he didn't respond within a week, better you fork the project.
FYI: There is a place called jazzband, and I think we can move this project there and which will help to get more exposure. There are some guidelines to make it work, jazzband guidelines, please have a look.
@GitRon
@jerinpetergeorge I know, you were just in CC. But good idea with the personal email address! Thx!
And good idea for jazzband as well!
@jerinpetergeorge I wrote him. We'll see what happens.
Any luck with this, I echo the use case that chunks being uploaded to S3 directly would be preferential.
@cr0mbly I am not sure if this is possible and wanted. S3 is not a file system but an object storage. So you'd expect the full file as an object, not some bunch of chunks.
Furthermore: What happens if something goes sideways during the upload? Then you have to clean up some leftover chunks?
IMHO the local temp approach is the cleaner solution. 😃
@cr0mbly I am not sure if this is possible and wanted. S3 is not a file system but an object storage. So you'd expect the full file as an object, not some bunch of chunks.
Yep thanks for this. I just put some work in on my own to integrate S3 as the default storage and came up against the fact that we can't append file to an S3 object. This is fine for small files as you can just keep updating the file during the backend but when you get larger files loading and downloading the full object into memory becomes less and less preformative.
Just for anyone else looking at this with autoscaling setup we left it as tempstorage and used EFS to maintain persistence of the filesystem during chunk upload.
If you use AWS S3 the default storage is not working. The chunks cannot be created and updated.
I added the possibility to activate cross-platform temporary storage. I cannot use
CHUNKED_UPLOAD_STORAGE_CLASS
because I need to pass thelocation
paramter to the storage.This is not working.
I decided not to change the way
CHUNKED_UPLOAD_STORAGE_CLASS
works for better compatibility.