search

ImperialCollegeLondon / django-drf-filepond

A Django app providing a server implemention for the Filepond file upload library
BSD 3-Clause "New" or "Revised" License
105 stars 40 forks source link

Add support for large files (>2 GB) #57

Closed iver56 closed 3 years ago

iver56 commented 3 years ago

offset and total_size in bytes are currently represented by int32 fields in the database:

https://github.com/ImperialCollegeLondon/django-drf-filepond/blob/dca2e3aeeacac60b6a24f2b83ab1d929f31d837c/django_drf_filepond/models.py#L140

This effectively limits the max file size to roughly 2 GB.

In my use case, django-drf-filepond sometimes needs to be able to swallow/accept large files. It would be nice if django-drf-filepond could use BigIntegerField (64 bits) instead of IntegerField (32 bits).

jcohen02 commented 3 years ago

Thanks for highlighting this @iver56. Happy to make the switch to BigIntegerField for both offset and total_size.

I'll leave last_chunk as an IntegerField since I can't really see a case where you'd need to have 2^32 chunks, you'd presumably hit other limitations before getting there anyway (e.g. filesystem limits)!

jcohen02 commented 3 years ago

Fix merged into master. I'm due to create a 0.3.1 patch release once some issues with the CI infrastructure are sorted out. Given that this introduces a database change, I'm inclined to include this in a 0.4.0 release so that anyone who wants to avoid the DB change can stick with the 0.3.x releases.

iver56 commented 3 years ago

Thanks for the swift and positive response :) Yes, it makes sense to include it in the 0.4.0 release given that it contains a DB migration

iver56 commented 3 years ago

Do you know roughly when 0.4.0 will be released? I'm asking because a colleague of mine proposes to cherry-pick the commit that fixes this (while we wait), but I feel like it's cleaner to wait and upgrade to 0.4.0 once it becomes available 😇

jcohen02 commented 3 years ago

Thanks for checking and apologies for the delay. I was actually hoping to also implement something for #51 in v0.4.0 since that is also likely to require a DB migration. However, I've not had a chance to look at that due to other commitments so I think it's probably best that I proceed with the v0.4.0 release to get the large file support in place and then push the other new features back to a later release. Just some final checks to go through and assuming all is looking OK, I'll aim to have 0.4.0 out and on PyPi within the next day or so.

jcohen02 commented 3 years ago

@iver56 release v0.4.0 with the large upload support is now out and available on PyPi. Any issues, let me know.

iver56 commented 3 years ago

Great! Thanks! I will give it a spin :)