FileUploaded is often not firing properly when uploading to Amazon

[emirotin]

We are going to use your plugin for a number of big projects for famous vendor. Currently we experiment using the free license, but in case of success we'll buy an OEM license.

The plugin is embedded into an upload control, and the testing page can be found here: http://seredov.flux-dev.com/GT/widgets-dev.html, this uses the uncompressed JS, so you can see how the Plupload is instantiated and events are bound.

When you click somewhere inside the widget it asks for the authorization, you can create an account. Then please use any small video file, upload it - you should see the blue progress bar below the file name.

I have to bind to the FileUploaded event because I need the server response for some pieces of information about the file registered on the server side. The file is uploaded to Amazon, then on server response I send an AJAX request to our server finalizing the upload, and then after the success the video details editing screen is shown.

Sometimes it works, but more often the upload is stuck (often at 100%, sometimes somewhere in the middle).

Sample Amazon HTTP response in case of "success"

HTTP/1.1 400 Bad Request
x-amz-request-id: 119A2032E4DA3376
x-amz-id-2: qiB5BVVChWR5SR+lh5Jjp+5X5hqZIKfi8GyEh0I2iAIyj5B7sOUMHPAILEGXK2xX
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Mon, 19 Dec 2011 08:30:58 GMT
Connection: close
Server: AmazonS3

15c
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>119A2032E4DA3376</RequestId><HostId>qiB5BVVChWR5SR+lh5Jjp+5X5hqZIKfi8GyEh0I2iAIyj5B7sOUMHPAILEGXK2xX</HostId></Error>
0

More often it says

HTTP/1.1 504 Fiddler - Receive Failure
Content-Type: text/html; charset=UTF-8
Connection: close
Timestamp: 11:44:11.614

ReadResponse() failed: The server did not return a response for this request.       

(as you see, I'm using Fiddler to inspect requests and responses, but it's clearly reproducible with Fiddler switched off).

Any help is appreciated as this issue is critical for Plupload adoption.

[jayarjo]

This is strange. As far as you are using 'success_action_status' => '201' (and I see in the code, that you are) and do not use chunking, you should not have any freeze problem at the end of the file. So far couldn't repeat freeze problem in monitored environment, although seemingly had one freeze in the beginning, before I switched on the sniffer.

Does it happen to you consistently across browsers, systems and workstations?

[jayarjo]

How big are the files that you upload?

[emirotin]

Thanks for the reply It usually goes smoothly with 1M, but constantly hangs for 23M gets file I think it's stably reproducible on different platforms / browsers - we have dedicated QA engineers.

[jayarjo]

23 mb is a quite small file. Never heard RequestTimeout happening to something less then couple of gbs. Hm...

I will try some more.

[emirotin]

Hi again. Don't want to make any pressure, but any chance we can have news by Monday? Various guys want to play with the uploader and see if it's production ready, and the issue with upload freezing is the only remaining blocker.

Is it reproducible for you? Do you need any help / comments on the widget functionality? You can also create your own testing page if you wish - simply copy the whole HTML from the page I referenced in the ticket, and can disable phone / agreements fields to simplify testing.

[jayarjo]

We are going to do a bugfix release today. Be sure to try it out.

[emirotin]

Oh, great to know, thank you

[jayarjo]

Not able to publish on the site at the moment, some technical obstacles. Will do that tomorrow. But you can download it from Downloads section, here on github.

Let us know of results :)

[emirotin]

Good I didn't notice any relevant changes to Flash in recent commits, but the 1st problem (upload freeze in the middle) is solved, and the 2nd problem (no response after 100%) looks like our local environmental issue (at least not reproducible from couple of external machines)

So, I'm closing the issue, and in case of new problems or details will open a new one.

Thank you for the help.

[jayarjo]

Thanks for testing it out and coming back with feedback. We are preparing a release for next major version of Plupload now, which is going to support checksum calculation and retry logic for chunked uploads. And is going to be much more compatible with Amazon S3 and other kind of CDNs.

[emirotin]

Nice to hear We'll update as soon as it's available and will repeat our tests from the local network

[emirotin]

Looks like it's reproducible again From what I see in Wireshark, Amazon responds with HTTP/1.1 400 Bad Request Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed

The testing env is http://seredov.flux-dev.com/GT/widgets-dev.html You can check the Plupload config by debugging the Upload script

browse_button "pickFileF13273268091982" chunk_size 0 container "uploaderContainerF13273268091981" file_data_name "file" filters [Object { title="Video files", extensions="avi,dv,mov,qt,mpeg,mp4,3gp,asf,wmv,flv,f4v"}]
extensions "avi,dv,mov,qt,mpeg,mp4,3gp,asf,wmv,flv,f4v"
title "Video files" flash_swf_url "http://cus.flux-devstatic.com/Other/plupload/plupload.flash.swf" init Object { FilesAdded=function()} FilesAdded function() max_file_size undefined
multi_selection false
multipart true
multipart_params AWSAccessKeyId "1YK82GxxxxxxxP0TGT82"
Filename "5897b44e-d039-4981-a6ae-2ff5cc5bf6ed"
acl "private" key "5897b44e-d039-4981-a6ae-2ff5cc5bf6ed"
policy "eyJjbxxxxxxxloifQ=="
signature "6cJgiiwPxxxxxE8qEI4BexFLX5ds=" success_action_status "201"
page_url "/GT/"
runtimes "flash,html5,silverlight,gears" silverlight_xap_url "http://cus.flux-devstatic.com/Other/plupload/plupload.silverlight.xap" url "http://fluxdev.s3.amazonaws.com"

[emirotin]

• my Flash version is 11.1

[emirotin]

UPD: sometimes when using urlstream_upload: true it uploads much faster, doesn't hang (looks like), but doesn't produce the progress events

[jayarjo]

Thanks for letting us know about the progress on this.

I'm afraid the only mode we do have control over (for Flash) is the urlstream mode. Another one is default one and is completely locked (no access to raw file data, no custom headers - the only thing you could do if you fail in non urlstream mode is retry the whole file).

For coming version we build up on top of urlstream mode. I'm experimenting now to get the progress on it.

[jayarjo]

So you have excluded all possible local factors?

[emirotin]

Well, actually not. From outside of our work network files look like uploading, though quite slow, but this can be an issue of my ISP.

So, from my previous comments what is relevant:

• in our network the internet is quite good, but they say for big files the tail packets have lower priority (to prioritize serfing before downloads)
• with urlstream_upload sometimes it's much faster and doesn't hang, though not in 100% of cases

Sorry for the buzz.

[jhjwind]

experiencing the same issue.

[mfulton]

I am seeing a similar issue, not with Amazon, where it doesn't seem to be firing the FileUploaded or UploadComplete events at the end. The file upload is succeeding as far as the server is concerned, but no events other than UploadProgress. This is with the FLASH client. My log output looks like this:

[Refresh] [FilesAdded] File: id=p16trllr5sqa47mfn99a6f1fiv1, name=Go-Daddy-Hosting.jpg, size=270614, loaded=0, percent=0, status=QUEUED [QueueChanged] [Refresh] [Refresh] [StateChanged] STARTED [Refresh] [UploadProgress] File: id=p16trllr5sqa47mfn99a6f1fiv1, name=Go-Daddy-Hosting.jpg, size=270614, loaded=262144, percent=97, status=UPLOADING, target_name=p16trllr5sqa47mfn99a6f1fiv1.jpg Total: size=270614, loaded=262144, uploaded=0, failed=0, queued=1, percent=97, bytesPerSec=491828 [UploadProgress] File: id=p16trllr5sqa47mfn99a6f1fiv1, name=Go-Daddy-Hosting.jpg, size=270614, loaded=270614, percent=100, status=UPLOADING, target_name=p16trllr5sqa47mfn99a6f1fiv1.jpg Total: size=270614, loaded=270614, uploaded=0, failed=0, queued=1, percent=100, bytesPerSec=261716

If something triggers a REFRESH event after the upload, then it appears to know that the previous upload was completed. That is, the file just uploaded is removed from the queue, and the "add files" and "upload" buttons are back, so this seems to be purely a matter of the events not making it through to the Javascript side of things.

My code to create the uploader is:

    $("#pluploader").pluploadQueue({
        // General settings
        runtimes : 'flash,silverlight,html5,browserplus,gears,html4',
        url : "<?=$uploadpath?>" + "&galleryid=" + galleryID,
        max_file_size : '20mb',
        //chunk_size : '10mb',
        unique_names : true,

        // Resize images on clientside if we can
        //resize : {width : 320, height : 240, quality : 90},

        // Specify what files to browse for
        filters : [
            {title : "Image files", extensions : "jpg,gif,png"},
            {title : "Zip files", extensions : "zip"}
        ],

        // Flash settings
        flash_swf_url : '<?="/wp-content/plugins/" . $_REQUEST['plugname']?>/plupload/js/plupload.flash.swf',

        // Silverlight settings
        silverlight_xap_url : '<?="/wp-content/plugins/" . $_REQUEST['plugname']?>/plupload/js/plupload.silverlight.xap',

        // Post init events, bound after the internal events
        init : 
        {
            ChunkUploaded: function(up, file, info) 
            {
                // Called when a file chunk has finished uploading
                log('[ChunkUploaded] File:', file, "Info:", info);
            },

            Destroy: function(up)
            {
                log('[DESTROY]');
            },

            Error: function(up, args) 
            {
                // Called when a error has occured
                log('[error] ', args);
            },

            FilesAdded: function(up, files) 
            {
                // Callced when files are added to queue
                log('[FilesAdded]');

                plupload.each(files, function(file) 
                {
                    log('  File:', file);
                });
            },

            FilesRemoved: function(up, files) 
            {
                // Called when files where removed from queue
                log('[FilesRemoved]');

                plupload.each(files, function(file) 
                {
                    log('  File:', file);
                });
            },

            FileUploaded: function(up, file, info) 
            {
                // Called when a file has finished uploading
                log('[FileUploaded] File:', file, "Info:", info);
            },

            QueueChanged: function(up) 
            {
                // Called when the files in queue are changed by adding/removing files
                log('[QueueChanged]');
            },

            Refresh: function(up) 
            {
                // Called when upload shim is moved
                log('[Refresh]');
            },

            StateChanged: function(up) 
            {
                // Called when the state of the queue is changed
                log('[StateChanged]', up.state == plupload.STARTED ? "STARTED" : "STOPPED");
            },

            UploadComplete: function(up, files)
            {
                // Called when files where removed from queue
                log('[UploadComplete]');

                plupload.each(files, function(file) 
                {
                    log('  File:', file);
                });
            },

            UploadFile: function(up, file) 
            {
                // Called while a file is being uploaded
                log('[UploadFile]', 'File:', file, "Total:", up.total);
            },

            UploadProgress: function(up, file) 
            {
                // Called while a file is being uploaded
                log('[UploadProgress]', 'File:', file, "Total:", up.total);
            }

        }
    });

[jigarshah]

Same issue...Its not getting fired every-time...

[mhollas]

I am having the same problem. FilesUploaded does not fire when I upload direct to Amazon S3. I have tested on the beta 2 also to no result.

[tangentfairy-zz]

I am having the same issue, it works some of the time.