23 / resumable.js

A JavaScript library for providing multiple simultaneous, stable, fault-tolerant and resumable/restartable uploads via the HTML5 File API.
MIT License
4.65k stars 611 forks source link

Python Flask back-end: final file corrupted due to insufficient chunks testing to define upload completed #619

Open eanon opened 3 weeks ago

eanon commented 3 weeks ago

Relying on the provided python back-end example as starting point for my own one, I discovered that sometimes the final file on the server side was corrupted (e.g., a chunk could have been merged twice). So, debugging through well-placed print's, I finally understood that talking about chunks which could be uploaded in parallel, checking the existence of the chunks file was not enough to indicate that the entire upload was achieved; some chunks could be there but still in progress.

So, I changed this code block:

    # check if the upload is complete
    chunk_paths = [os.path.join(temp_dir, get_chunk_name(resumableFilename, x)) for x in range(1, resumableTotalChunks+1)]
    upload_complete = all([os.path.exists(p) for p in chunk_paths])

    # combine all the chunks to create the final file
    if upload_complete:

By this one:

        # check if all chunks fully downloaded
        chunk_paths = [os.path.join(temp_dir, get_chunk_name(resumableFilename, x)) for x in range(1, resumableTotalChunks+1)]
        chunks_total = 0
        chunks_size = 0
        for chunk_path in chunk_paths:
            if not os.path.exists(chunk_path):
                break
            chunks_total += 1
            chunks_size += os.stat(chunk_path).st_size

        # combine all the chunks to create the final file
        if (chunks_total == resumableTotalChunks) and (chunks_size == resumableTotalSize):

This way, we ensure that reassembling the chunks is only operated when all the chunks are here and terminated for sure. Well... Sorry, I've no time to provide a pull request, but I wanted to let you know through this quick report (feel free to use it in your example if you want).