Chunked File Uploads have different file size/corrupt

[markpolarisdesign]

Dear all,

I have a problem with the chunked parallel upload. If I set the parallelChunkUploads is false. file size the same as the original but very slowly uploading. If I set the parallelChunkUploads is true. It's not working. file size different but very fast uploading.

Original File:

• 30MB.tif - (30MB)

Chunk File:

• 30MB_1.tif-liD1YsLk2s3RBICinzOek6O3gaPfZYUj5QolFN7S-c181b1b31eef0b87d1c5753e59fe7e9e.part - (26MB)
• 30MB_1.tif-liD1YsLk2s3RBICinzOek6O3gaPfZYUj5QolFN7S-c181b1b31eef0b87d1c5753e59fe7e9e-0e4ecbfb-2827-488d-bdeb-4f0e74300e97.13.part - (2MB)
• 30MB_1.tif-liD1YsLk2s3RBICinzOek6O3gaPfZYUj5QolFN7S-c181b1b31eef0b87d1c5753e59fe7e9e-0e4ecbfb-2827-488d-bdeb-4f0e74300e97.14.part - (2MB)

Merging is not complete. It should be only a single file with 30MB as same as the original.

JavasScript

function initDropzone(directory)
    {
        let totalTime, startTime, endTime, prevPercent = 100;
        let dataFileDropzone = new Dropzone("#upload-datafile-file-"+directory, {
            autoProcess: true,
            chunking: true,
            method: "POST",
            maxFilesize: 10485760000,
            chunkSize:  2 * 1024 * 1024,
            timeout: 216000000,
            acceptedFiles: ".zip,.rar,.tar,.bz,.jpeg,.jpg,.tiff,.tif,.bz2,.hdf5,netcdf,.hrit,.lrit,.gif,.png,.txt,.csv,.xls,.pdf,.doc,.docx,.ppt,.pptx",
            parallelChunkUploads: true,
            dictDefaultMessage: 'Drop files here or click to upload',
            success: function(file, responseText, data) {

                    let responseJSON = JSON.parse(file.xhr.response);
                    console.log(file);
                    endTime = moment();
                    totalTime = moment.duration(endTime.diff(startTime));
                    console.log(totalTime.asSeconds())
                    $('#datafileTable-'+directory).append(datafileTemplate(responseJSON));

                    axios({
                        method: 'post',
                        url: '/ajax/v1/data_provider_node/'+DPNID+'/logs',
                        params: {
                            organization: responseJSON.organization,
                            directoryPath: responseJSON.file,
                            fileName: responseJSON.full_name,
                            fileSize: file.size,
                            executeTime: totalTime
                        }
                    }).then(function (resp) {
                        console.log(resp)
                    }).catch(function (error) {
                        console.log('Write log error')
                    });

            },
            removedfile: function(response) {
                console.log('File Removed');
            },
            queuecomplete: function(response) {
                console.log('Queue Completed');
            }
        });
        dataFileDropzone.on('uploadprogress', function(progress, bytesSent) {
            console.log(progress, bytesSent );
        });

        // Append token to the request - required for web routes
        dataFileDropzone.on('sending', function (file, xhr, formData) {
            formData.append('_method', 'PUT');
            formData.append("_token", $('meta[name="csrf-token"]').attr('content'));
            formData.append("directory", directory);
        });
        //
        dataFileDropzone.on('complete', function (progress) {
            console.log(progress);
        });

Controller

public function uploadDatafile(DPN $DPN, Request $request)
    {
        $receiver = new FileReceiver("file", $request, HandlerFactory::classFromRequest($request));
        $destination = $DPN->emergency_request->key . '/' . $request->input('directory') . '/' . $DPN->organization->abbreviation;

        if ($receiver->isUploaded() === false) throw new UploadMissingFileException();

        if ($receiver->isUploaded() === true) $save = $receiver->receive();

        if ($save->isFinished())  $this->saveFileDatafile($save->getFile(), $destination, $DPN);

        $handler = $save->handler();

        return response()->json([
            "done" => $handler->getPercentageDone(),
            'status' => true
        ]);
}

protected function saveFileDatafile(UploadedFile $file, $destination, $DPN)
    {
        $extension = $file->getClientOriginalExtension();
        if(in_array($extension, $this->dangerousExtension)) {
            return response()->json(false);
        }
        $fileName = DPN::cleanFileName($file->getClientOriginalName());
        $this->storage = Storage::disk(env('STORAGE_DISK_PAST'));
        $this->storage->putFileAs($destination, $file, $fileName);

        return response()->json(true);
    }

Best regards, Mark

[pionl]

Fixed a bug and added a support for Laravel 7 in v1.4.0 that will fix your issues.

🎉 Finally a release for Laravel 7 support ⛑ Finally I've fixed problems with chunks (Dropzone/Resumable.js with parallel upload). 👍 Also I've found a solution how to get file data after upload for a Dropzone. Check WIKI.

Thank you for your patience.

[markpolarisdesign]

@pionl Thank you so much for your time to fix this problem. I will revert my script to parallel upload and try it again.