Web Based File Upload Chunking

[matart]

Steps to reproduce

Web based large file upload

Expected behaviour

File gets chunked and multiple ajax requests send the chunks to the server for reassembly

Actual behaviour

File is uploaded as a whole file. A single request is made

Server configuration

Operating system: RHEL 6

Web server: NGINX

Database: MySQL

PHP version: 5.5

ownCloud version: 6.0.3

Updated from an older ownCloud or fresh install:

List of activated apps: Files, Activity, Contacts, Pictures

The content of config/config.php:

I had to remove the identifying URLs so it seems a little empty.

<?php
$CONFIG = array (
  'instanceid' => 'oc8fdcdb7be6',
  'passwordsalt' => '',
  'datadirectory' => '/var/www/html/macdrop_assets',
  'dbtype' => 'mysql',
  'version' => '6.0.3.1',
  'dbname' => '',
  'dbhost' => '',
  'dbtableprefix' => '',
  'dbuser' => '',
  'dbpassword' => '',
  'installed' => true,
  'forcessl' => true,
  'theme' => 'macdrop',
  'proxy' => '',
  'loglevel' => '2',
  'maxZipInputSize' => 1073741824,
  'allowZipDownload' => true,
  'mail_domain' => '',
  'mail_smtphost' => '',
  'mail_smtpname' => 'macDrop',
  'maintenance' => false,
  'trusted_domains' => 
  array (
    0 => 'url1',
    1 => 'url2',
  ),
);

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: yes/no

Client configuration

Browser: Chrome

Operating system: Windows

Browser log

This is a 1.7GB file. One request taking 3.2 minutes.

If this feature does not exist I am willing to look into implementing it and making a pull request.

[karlitschek]

I think this feature doesn´t exist at the moment. So a pull request would be awesome. @butonic Haven´t you looked into that last year?

[ghost]

AFAIK there is already a discussion / feature request for chunked file uploading available at the bugtracker but i can't find it at the moment. Will see if i can dig this out tomorrow.

[matart]

That would be great. Thank you! Thank you,Mathew A. From: RealRancorSent: Monday, June 9, 2014 4:53 PMTo: owncloud/coreReply To: owncloud/coreCc: Mathew ArtemenkoSubject: Re: [core] Web Based File Upload Chunking (#8955)AFAIK there is already a discussion / feature request for chunked file uploading available at the bugtracker but i can't find it at the moment. Will see if i can dig this out tomorrow.

—Reply to this email directly or view it on GitHub.

[PVince81]

I believe I heard that some app, maybe this one http://apps.owncloud.com/content/show.php/Large+File+Upload?content=159291 was using HTML5 file chunking of some sort.

[tylla]

Yes that app was working until OC 8.0. Now it is defunct and the maintainer has some problems fixing it. I believe too that this functionality should be implemented as a core functionality, so someone with enough knowledge should take a look at that app's code and merge it to the core. Unfortunately I'm not that person.:(

[PVince81]

Yes, would be a nice addition.

Note that there are plans to make the web UI use WebDAV first: https://github.com/owncloud/core/issues/12353

This means it will become possible to use chunking too through the WebDAV endpoint.

[tylla]

As I found out, OC 8 uses jQuery-File-Upload and as I discovered here jQuery-File-Upload already supports chunked uploading. One only has to override the maxChunkSize variable with some sane value instead of the default "undefined", and after that the upload starts to send chunks with the specified size instead of whole files. For testing I changed to maxChunkSize: 1000000, // 1MB

One problem exists though. At the end of the upload there is an error message that asks whether I want to overwrite the same file with the same file. If it's gibberish what I say, there is a screenshot attached. There you can see that the already uploaded image is only the first chunk (there is some missing data from the image too) and asks me to overwrite it with the full file. But if I choose the newer or both files it just hangs after I click the Continue button. After this I can click the Cancel, but then I'm also only with the partial file. BTW it throws a console error when I click the Continue button: TypeError: data.formData.push is not a function data.formData.push({name:'resolution', value:'autorename'}); //hack for ie8 file-upload.js?v=8ca31e8dc30c45c0bf2c0abdfb1746bd (line 155, col 42)

Hmm, this starts to look as a bug report... maybe I should file it as such.

[PVince81]

But also I believe that the ajax/upload.php endpoint does not support chunks, so it probably won't work. Ideal would be to use the WebDAV endpoint instead. Note that the WebDAV endpoint uses custom chunking (so does the sync client), so it might not work either with a "standard" solution.

Either way, it will need a bit of work to make this.

Still, any information you post here could be useful for that, so thanks :smile:

[PVince81]

I guess it asks you for overwrite because for each chunk sent to upload.php, it saves it as a new file. Because the ajax/upload.php endpoint doesn't understand chunks. And since the file exists already, it shows you that dialog. (just guessing)

[PVince81]

PR here https://github.com/owncloud/core/pull/26306

[PVince81]

new/updated PR here: https://github.com/owncloud/core/pull/28415

[PVince81]

Reopening this to address the "public page upload" which cannot use chunking currently due to API limitations: basically the chunking API only works for authenticated clients and needs to be enhanced.

[PVince81]

Requires https://github.com/owncloud/core/issues/23269 to provide a public DAV server that supports multiple subpaths like "public.php/dav/uploads".

[hodyroff]

People can use the guest app. Receiving chunked uploads via webfrontend seems not a P1 priority, adjusting. Please add your vote here if you need this.

[PVince81]

What we need to make anonymous upload support chunking:

• [ ] New Webdav endpoint for public file access: https://github.com/owncloud/core/issues/23269
• [ ] Adjust the "uploads" Webdav endpoint to also allow anonymous users to use: I suggest using the share token instead of user id. We also need to make sure that concurrent anonymous users cannot influence each other's uploads.
• [ ] Adjust public link page frontend to use the two points above for file access and uploads

[fritzmg]

I would love to see chunking implemented for public anonymous uploads. This way we can send customers or service partners simply a public link for a folder which has Upload only (File Drop) enabled and they can drop in their files there, which might be several Gigabyte in size, without having to create a guest account.

[hurradieweltgehtunter]

Since we advertise no upload limits on public uploads on certain projects I wanted to raise this topic.

[butonic]

cc @C0rby

[micbar]

This is a won't fix. We may have this in ownCloud infinite scale where we use tus.io as an upload protocol.