masterzen / nginx-upload-progress-module

Nginx module implementing an upload progress system, that monitors RFC1867 POST uploads as they are transmitted to upstream servers.
http://wiki.codemongers.com/NginxHttpUploadProgressModule
Other
432 stars 101 forks source link

Sometimes it's usefull to watch several simultaneous uploads in one request. Especially for a big number of such uploads. #30

Open devgs opened 12 years ago

devgs commented 12 years ago

Sorry if it looks hacky for you.

thomascrown commented 12 years ago

How would I call this in a form and what javascript would I use to get multiple status for a particular unique ID? Do I have to give each upload a unique ID? Do you have an example of the config file you used? This looks exciting if I could get it working!!!

devgs commented 12 years ago

How would I call this in a form and what javascript would I use to get multiple status for a particular unique ID?

Nope, you perform several uploads simultaneously when each has unique ID.

Do I have to give each upload a unique ID?

Yes.

Do you have an example of the config file you used?

Nothing special

http {

upload_progress proxied 1m;

server {

   listen 127.0.0.1:80;
   location /progress {
      report_uploads proxied;
      upload_progress_json_multiple_output;
   }

   location ~ /upload/.* {
                access_log  /var/log/nginx/access.log main;
                proxy_pass  http://file-backend;

                proxy_redirect     off;

                proxy_set_header   Host             $host;
                proxy_set_header   X-Real-IP        $remote_addr;
                proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
                proxy_intercept_errors on;

                client_max_body_size       0;
                client_body_buffer_size    512k;

                client_body_temp_path      /tmp/nginx/client_body_temp;

                proxy_connect_timeout      120s;
                proxy_send_timeout         120s;
                proxy_read_timeout         120s;

                proxy_buffer_size          256k;
                proxy_buffers              128 4k;
                proxy_busy_buffers_size    256k;
                proxy_temp_file_write_size 256k;

                track_uploads proxied 30s;
        }
}

This looks exciting if I could get it working!!!

To get upload status you knock /progress with header X-ProgressMultiple-ID set to <ID1>[;<ID2>[;<ID3>...]]

And you will get JSON array

[<json_multiple_pattern for ID1>, <json_multiple_pattern for ID1>, ...]

Default pattern is :

static ngx_str_t ngx_http_uploadprogress_json_multiple_defaults[] = {
    ngx_string("{ \"id\" : $uploadprogress_id, \"state\" : \"starting\" }"),
    ngx_string("{ \"id\" : $uploadprogress_id, \"state\" : \"error\", \"status\" : $uploadprogress_status }"),
    ngx_string("{ \"id\" : $uploadprogress_id, \"state\" : \"done\" }"),
    ngx_string("{ \"id\" : $uploadprogress_id, \"state\" : \"uploading\", \"received\" : $uploadprogress_received, \"size\" : $uploadprogress_length }")
’’’
};
thomascrown commented 12 years ago

@devgs Having a problem compiling your patch - keep getting build errors when running *make in function 'ngx_http_track_uploads 1723: error: 'NGX_PARSE_LARGE_TIME' undeclared 1723: error: (Each undeclared identifier is reported only once for each function it appears in

devgs commented 12 years ago

@thomascrown Mhm... I can't find (NGX_PARSE_LARGE_TIME) symbol there. Also line 1723 is not in the function ngx_http_track_uploads.

Maybe you are trying to compile different version of source code?

thomascrown commented 12 years ago

@devgs Yup - it was my bad - no idea what file I was trying to use - i just replaced the .c file with your changed version. It seams to be working - thanks for the help so far - I really appreciated it!

Now I have other problems: I can call the progress OK with X-ProgressMultiple-ID.... but it always says starting. My setup worked without the patch & with just one file using the jquery.uploadProgress.js extension....

Do you have a working example of a form with javascript that has multiple uploads working?

My config for Nginx server level is as follows:

location @frontcontroller { rewrite ^ /upload/index.php last; }

location = /progress { report_uploads uploads; upload_progress_json_multiple_output; }

location /upload { upload_pass @frontcontroller; upload_store /var/www/tmp_upload 1; upload_store_access user:r group:r all:r; upload_set_form_field $upload_field_name.name "$upload_file_name"; upload_set_form_field $upload_field_name.path "$upload_tmp_path"; upload_aggregate_form_field "$upload_field_name.sha1" "$upload_file_sha1"; upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size"; upload_pass_form_field "^target$"; upload_cleanup 400 404 499 500-505; track_uploads uploads 5s; }

I cannot thank you enough for your help!!!

SigSegFault commented 12 years ago

@thomascrown That's me (devgs). Sorry for posting from different account.

From what i read in your doc: ''' Since Nginx doesn't support yet RFC 1867 upload, the location must be a proxy_pass or fastcgi location.'''

upload_pass will not work. It says 'starting', because when there is no such ID found, you return such status.

I hope that will help.

So far i can't provide example because i am back-end developer and have poor knowledge of JS :(

But soon we are going to use this in production and if you wish, I can give link you our website so you can test it with firebug.

thomascrown commented 12 years ago

@devgs @SigSegFault Thanks for the help - I do have the front end working - and the setup that I pasted above works when I am using the non patched version of progress upload...

In any case - I created a wrapper that accept dropped files, and that passes each file to the upload processor with a unique ID for each... it does the trick and Im getting accurate load times.

Once again - many thanks for the help!

devgs commented 12 years ago

Happy to help ;) Thank you.

masterzen commented 11 years ago

I know this is a long time with no answers on this PR, but can you please squash those commit in a more coherent patch set.

Oh, and also move out the graphite commit in its own PR out of this one would be really good.

Also it would be great to support the multiple ids with the exact same API as the single one (ie send the same headers, but detect when more than one id is sent and thus return a json array instead of a single json thing).

Thanks,