Feature request: Add support for chunkUploads

[iver56]

Thanks for making django-drf-filepond!

When I try to enable chunkUploads in the filepond client, the process view in django-drf-filepond raises an exception like this:

Internal Server Error: /fp/process/
Traceback (most recent call last):
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django\views\decorators\csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django\views\generic\base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\rest_framework\views.py", line 505, in dispatch
    response = self.handle_exception(exc)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\rest_framework\views.py", line 465, in handle_exception
    self.raise_uncaught_exception(exc)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\rest_framework\views.py", line 476, in raise_uncaught_exception
    raise exc
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\rest_framework\views.py", line 502, in dispatch
    response = handler(request, *args, **kwargs)
  File "C:\Users\Iver\Anaconda3\envs\my_project\lib\site-packages\django_drf_filepond\views.py", line 130, in post
    upload_filename = file_obj.name
AttributeError: 'str' object has no attribute 'name'

This made me realize that django-drf-filepond currently does not support chunked uploads. Here is a description of how filepond treats chunked uploads (the initial POST request does not include a file):

https://pqina.nl/filepond/docs/patterns/api/server/#process-chunks

Are there any plans for adding support for chunked uploads in django-drf-filepond?

[iver56]

Chunked uploads really help when uploading large files over unreliable connections.

This piece of code shows how to add a chunk to an existing partial file: https://github.com/juliomalegria/django-chunked-upload/blob/2d49a8c35bc60570c0ced68164c3fee34d312f33/chunked_upload/models.py#L63

I believe some of that can be reused when implementing chunked uploads

[jcohen02]

Hi @iver56, thanks for this feature request. I can see that handling chunked uploads looks like it could be a useful enhancement.

I'll take a look at this in the next couple of days and see what would be involved in adding this and hopefully this is something that can be included in the next release.

[ayushpalak]

Hi @jcohen02 ,

Thanks for working on chunk upload feature. May I know when will this feature will be released?

[jcohen02]

Hi @ayushpalak, sorry for the delay in implementing this but thanks for the reminder. I've been caught up with other things over the last few weeks but have now started work on this feature and hope to have something available very soon. (likewise, apologies @iver56 for the delay).

[jcohen02]

An initial, fully functional, implementation of chunked uploads has just been pushed to the feature/chunked_upload branch.

There's still a bit of work to do here in terms of completing the updates to the test suite, documentation etc, however I wanted to get the core implementation out as soon as possible in case anyone wants to test this and provide any feedback at this stage.

With this branch deployed, it should be possible to set the chunkUploads property on your filepond client configuration to true and set chunkSize, as described in the filepond docs and then also add a new patch endpoint to the server block in the configuration, so you'll have something like:

chunkUploads: true,
chunkSize: 1048576,
server: {
    ...
    patch: '/patch/',
    ...
}

This should then correctly handle files of greater than chunkSize in multiple chunks. If the transfer fails after multiple retries of a chunk, it should also correctly pick up the transfer again following the last successfully transferred chunk when the retry button is clicked in filepond UI.

I'm hoping to have this finalised with documentation and a full set of tests early next week. I'll add an update to this issue when that's done.

[iver56]

I tried the implementation of chunked uploads in the feature/chunked_upload branch, and it seems to work well for me so far. Thanks! 😄🥇

[jcohen02]

Great, many thanks for checking this out @iver56. Still finalising things with tests and docs here but aiming to get this merged in and into a new release by the of this week.

[jcohen02]

v0.3.0 including chunked upload support has just been released.

Thanks @iver56 for the original feature request and for trying out the initial implementation.

I'll leave this issue open for a little while in case there are any problems relating to the new functionality.

[jcohen02]

Closing this now since this functionality has been in place for a little while and all seems ok.