Multipart Support

[moshir]

Hi,

For very large files, I doubt sending the whole data in the request body will make it for my use case. i'm trying to understand how I could hack the ExtJsUploader class so that multipart requests are sent. Find myself a little bit lost in the Ext.data.Connection docs (http://www.objis.com/formationextjs/lib/extjs-4.0.0/docs/api/Ext.data.Connection.html).

Seems hard to send multipart requests from ext without using forms. Do you have any idea how this could be supported. Is it possible to wrap Items in form components ?? Any idea appreciated.

[ivan-novakov]

Hi, in Chrome I was able to upload 2 GB file without problems. Why do you require the multipart variant? Maybe your backend doesn't support the raw POST? I chose the raw POST method, because it's easy and straightforward. There is no need to twist the specifications. And it works with the File API. To send multipart requests you need to simulate form submission (I guess). But in that case I'm not sure if it will be compatible with mine solution...

Maybe you could try alternative solutions: http://www.sencha.com/forum/showthread.php?205576-File-upload-with-drag-amp-drop-support&highlight=upload

[moshir]

I'm using node.js and seems like long body cannot be parsed. I have no issues with File API but I'm not sure how it works, is all the file content read in memory at once, or is there some kind of streaming process ?

On Wed, Jan 16, 2013 at 2:46 PM, Ivan Novakov notifications@github.comwrote:

Hi, in Chrome I was able to upload 2 GB file without problems. Why do you require the multipart variant? Maybe your backend doesn't support the raw POST? I chose the raw POST method, because it's easy and straightforward. There is no need to twist the specifications. And it works with the File API. To send multipart requests you need to simulate form submission (I guess). But in that case I'm not sure if it will be compatible with mine solution...

Maybe you could try alternative solutions:

http://www.sencha.com/forum/showthread.php?205576-File-upload-with-drag-amp-drop-support&highlight=upload

— Reply to this email directly or view it on GitHubhttps://github.com/ivan-novakov/extjs-upload-widget/issues/3#issuecomment-12318952.

Moshir MIKAEL Portable : 06 22 05 27 67

[moshir]

By the way, you mention the File API, but how do you read the file content. I see no occurence of the filereader read methods (readAsBuffer, readAsText ...). Do you delegate this to Extjs ?

On Wed, Jan 16, 2013 at 3:05 PM, moshir mikael moshir.mikael@gmail.comwrote:

I'm using node.js and seems like long body cannot be parsed. I have no issues with File API but I'm not sure how it works, is all the file content read in memory at once, or is there some kind of streaming process ?

On Wed, Jan 16, 2013 at 2:46 PM, Ivan Novakov notifications@github.comwrote:

Hi, in Chrome I was able to upload 2 GB file without problems. Why do you require the multipart variant? Maybe your backend doesn't support the raw POST? I chose the raw POST method, because it's easy and straightforward. There is no need to twist the specifications. And it works with the File API. To send multipart requests you need to simulate form submission (I guess). But in that case I'm not sure if it will be compatible with mine solution...

Maybe you could try alternative solutions:

http://www.sencha.com/forum/showthread.php?205576-File-upload-with-drag-amp-drop-support&highlight=upload

— Reply to this email directly or view it on GitHubhttps://github.com/ivan-novakov/extjs-upload-widget/issues/3#issuecomment-12318952.

Moshir MIKAEL Portable : 06 22 05 27 67

Moshir MIKAEL Portable : 06 22 05 27 67

[moshir]

Think i've got something working to support the multipart API, while keeping the file API. All changes take place in the ExtjsUploader file. First, replace the initConnection

    initConnection : function(){
         // simple javascript xhr object
         xhr = new XMLHttpRequest();
         xhr.open("post", "/api/upload", true);
         return xhr   ;
    }

Next, in the uploadItem code, the idea is to build the form and send its data as a simple xhr request :

var file = item.getFileApiObject();
        item.setUploading();
        var formData = new FormData();
        formData.append(file.name, file);

        var xhr = this.initConnection();
        // set headers
        xhr.setRequestHeader("X-File-Name", file.name);
        xhr.setRequestHeader("X-File-Size", file.size);
        xhr.setRequestHeader("X-File-Type", file.type);

        // set handlers
        var successhandler =  Ext.Function.bind(this.onUploadSuccess, this, [
            item
        ], true);

        var progresshandler = Ext.Function.bind(this.onUploadProgress, this, [
            item
        ], true);

        var failurehandler = Ext.Function.bind(this.onUploadFailure, this, [
            item
        ], true);

        xhr.upload.addEventListener("progress", progresshandler, true);
        xhr.upload.addEventListener("load",successhandler , true);
        xhr.upload.addEventListener("error", failurehandler, true);

        // send the form
        xhr.send(formData);
    }

The thing is i'm not sure it's totally comptaible with all the plugin pieces. It seems that once the upload button was clicked, you cannot browse for new files. Any clue ?

[ivan-novakov]

I'm passing the file reference as a "xmlData" parameter to the request call:

http://docs.sencha.com/ext-js/4-1/#!/api/Ext.data.Connection-method-request

Ii think, that different browsers handle it in different ways. Mozilla Firefox tries to read the whole file into memory and then sends it. Chrome "streams" it on the fly.

[ivan-novakov]

As for your suggestion - does it really perform the upload? What kind of object is the FormData object? Is there any working example?

[moshir]

Yes, it performs the upload. I receive the data server side (node.js with express framework). The formdata object is a browser object, there's no dependency here. I guess it holds the file data though i'm not positive about it. If you change #.initConnection and #.uploadItem methods like suggested in my revious post, it might work.

[rhalff]

Because you send the data as xmlData, the body will be of the type 'Xml Document text' (if I recall it correctly).

So I guess the uploader works accidently in your situation (maybe php understands it's not Xml Document text.

I also had to hack the connection to make this code work, anyway the rest of the code is great :-)

This is the hack I used:

/**
 * Implements {@link Ext.ux.upload.uploader.AbstractUploader#uploadItem}
 * 
 * @param {Ext.ux.upload.Item} item
 */
uploadItem : function(item) {
    var file = item.getFileApiObject();
    if (!file) {
        return;
    }

    item.setUploading();

    this.conn = this.initConnection();

    var me = this;

    var reader = new FileReader(); //Create FileReader object to read the image data
    reader.readAsDataURL(file); //Start reading the image out as binary data
    reader.onload = function() { //Execute this when the image is successfully read
    me.conn.request({
        scope : me,
        headers : me.initHeaders(item),
        rawData : reader.result,

        success : Ext.Function.bind(me.onUploadSuccess, me, [
                item
            ], true),
        failure : Ext.Function.bind(me.onUploadFailure, me, [
                item
            ], true),
        progress : Ext.Function.bind(me.onUploadProgress, me, [
                item
            ], true)
     });
   }
}, 

It will send the data to the server in this form (base64 encoded):

Accept:/ Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 Accept-Encoding:gzip,deflate,sdch Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4 Connection:keep-alive Content-Length:2940403 Content-Type:application/x-www-form-urlencoded Host:localhost:3000 Origin:http://localhost:3000 Referer:http://localhost:3000/reditor/ User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22 X-File-Name:c081259.jpg X-File-Size:2205284 X-File-Type:image/jpeg X-Requested-With:XMLHttpRequest

Form Data:

data:image/jpeg;base64,/9j/4AAQSkZJRgA......

There are probably a lot more ways to do it, but this works for me. I use nodejs on the server side so I have a lot of control on how to receive the request.

[ivan-novakov]

I guess, I used the xmlData property because there was no other option (except jsonData, which is similar) how to post data directly. And I wanted to keep the code as consistent with ExtJS as possible. Now I noticed, that in version 4.2 there is a new "biinaryData" property:

http://docs.sencha.com/extjs/4.2.1/#!/api/Ext.data.Connection-method-request

Anyway, using the FileReader API seems to be more elegant, so I'll try to find some time and update the code. Thanks!

[rhalff]

Mine is not very generic also, I just modified it so it works for me.

I have actually changed it again to:

     jsonData :{  
              image: reader.result
                        .substring(reader.result.indexOf('base64,') + 7)
                        // remove data:image/png;base64,
            },

And I send it as application/json, probably the code from moshir is a more normal way to send it.

Both in my case and in moshir's case the extra X-File-* headers are not really necessary I think, it's already part of the form data (although I didn't try the mulipart code) and in my case I could put it inside the JSON.

At the moment my server side expects the PUT requests to be JSON, so this was the laziest thing to do.

Another option could be to just send the image itself, PUT it as content type image/jpeg and all other headers that come with an image.

So many options, but I guess this is why you created the AbstractUploader in the first place :-)

To be complete, the server side nodejs code looks like this:

exports = module.exports = function(options) {

  var options = options || {};

  return function upload(req, res, next) {

    if(!req.body || !req.body.image) {
      console.log('Upload error req.body.image not found');
      return next();
    }

    if(typeof req.headers['x-file-type'] == 'undefined') return next();

      // we use express.json(), so our json is in req.body now.
      var contentType = req.headers['x-file-type'];
      var path = req.headers['x-file-name'].replace(/^\//, ''); // remove leading /
      var fullpath = req.headers['x-itemid'] + '/' + path;

      var imgBuffer = new Buffer(req.body.image, 'base64');

      sendToAmazonS3(res, path, fullpath, contentType, imgBuffer);

  };

};

[ivan-novakov]

rhalff,

I tried your approach with the FileReader and it doesn't work well with larger files. Under Chrome 25 the page crashes. Under Firefox 21 it hangs. At the same time, if I pass the File object directly, the widget is able to upload files > 1GB with no problems.

On the other hand, passing the File object directly as a rawData option seems a bit like magic to me. But it seems that it is intended and as far as I understand the specifications, the xhr.send() call accepts data argument of the type Blob and the File interface inherits from this interface.

http://dev.w3.org/2006/webapi/FileAPI/#blob https://dvcs.w3.org/hg/xhr/raw-file/tip/Overview.html#the-send()-method

[ivan-novakov]

moshir,

I tested your proposal to use the FormData object and it works fine. I will implement an alternative FormDataUploader so the widget will support both the raw POST and multipart ways.

[ivan-novakov]

I created the FormDataUploader in the devel branch. Feel free to test it. Now you can "inject" the uploader into the upload panel, which is now separated from the dialog, see:

https://github.com/ivan-novakov/extjs-upload-widget/blob/devel/public/app.js#L39