NginxHttpUploadProgressModule received and size are the same

[mhf-ir]

i have same issue with latest version of nginx using dotdeb.org repository deabian wheezy https://www.ruby-forum.com/topic/204982

new Object({ 'state' : 'starting' }) new Object({ 'state' : 'uploading', 'received' : 4505554, 'size' : 4505554 }) new Object({ 'state' : 'uploading', 'received' : 4505554, 'size' : 4505554 }) new Object({ 'state' : 'uploading', 'received' : 4505554, 'size' : 4505554 }) new Object({ 'state' : 'uploading', 'received' : 4505554, 'size' : 4505554 }) new Object({ 'state' : 'uploading', 'received' : 4505554, 'size' : 4505554 }) new Object({ 'state' : 'done' })

nginx version: nginx/1.4.4 TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --with-pcre-jit --with-debug --with-file-aio --with-http_addition_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module --with-http_gzip_static_module --with-http_gunzip_module --with-http_image_filter_module --with-http_mp4_module --with-http_perl_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_spdy_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-mail --with-mail_ssl_module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/headers-more-nginx-module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-auth-pam --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-cache-purge --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-dav-ext-module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-development-kit --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-echo --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/ngx-fancyindex --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-push-stream-module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-lua --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-upload-progress --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-upstream-fair --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-syslog --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/ngx_http_pinba_module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/ngx_pagespeed --add-module=/usr/src/nginx/source/nginx-1.4.4/debian/modules/nginx-x-rid-header --with-ld-opt=-lossp-uuid

[masterzen]

How are you testing? You're sending 4MB, which if tested over localhost won't show any progress because it is way too fast. Add some throttling if you want to see something useful.

If that's over a slow link, please recompile nignx with --with-debug and send me the error.log so that I can track the problem.

[mhf-ir]

It's not on localhost. i will try it on debug mode. i see this error on production mode :( on web page. I'm using debian and dotdeb repository that have latest version of nginx-extras package

[masterzen]

I was able to reproduce the issue, it's because of a change in recent nginx versions that doesn't track the remaining count of bytes to be uploaded (the quantity is never decreased). I think I've almost found a workaround, I'll try to release that ASAP.

[masterzen]

As explained in #32, if you reduce client_body_buffer_size to a sensible value compared to what you upload, then the upload progress will work.

[pulse00]

@masterzen thanks for looking into this,

sensible value compared to what you upload

how would i determine what this sensible value is? our uploads are fairly large, ranging from something aroung 50M to 300M.

[masterzen]

@pulse00 first, make sure to run the latest version I just pushed yesterday evening (0.9.1) it will fix the issue, irregardless of the value of client_body_buffer_size.

Regarding client_body_buffer_size, this buffer is allocated by nginx on each connection that sends a body to the server (ie most POST requests in fact, not only upload ones). If you allocate this to a large number you risk to DOS your server (because not enough RAM if there are many incoming connections trying to POST something).

You need to find a balance between the allowed number of incoming connections and performances: if you upload something larger than the buffer, the buffer content is stored to a temporary file, if the upload is smaller than the buffer everything stays in RAM. For largest uploads I don't think performances will suffer if the uploads ends in a temporary file before being proxied/fastcgied.

The default value for client_body_buffer_size is around 16k for 64 bits servers. It all depends on what you can afford in terms of RAM. I'd say that it's not a big deal to slow down a bit the very large uploads by reducing this settings to something like 50M ~ 100M if you can afford 100M+ per connections to your servers. YMMV though, sometimes it might be more efficient to spool the upload on disks than allocate large amount of memory if memory is a scarce resource.

You can read this thread for some background information.

Hope that helped,

[technicalabuse]

Unfortunately, I'm seeing this issue with NGINX 1.9.10, http2 and pagespeed enabled. I am happy to provide whatever relevant debug information unless this is a known issue.

{ "state" : "uploading", "received" : 84672387, "size" : 84672387 }

[dschissler]

I'm seeing this with nginx 1.8.1 normal package on Ubuntu 14.04 without pagespeed or spdy (this version doesn't have http2).

[Ramel]

Same thing here with nginx/1.8.0 (ppa:justice-w/nginx-upload) on Trusty Ubuntu 14.04, strange received == size.

[dschissler]

@Ramel upgrade with this PPA. Then dealing with #41 is trivial.

[Ramel]

Tried with 1.9.11, compiled a deb to include the upload module (https://www.nginx.com/resources/wiki/modules/upload/). But still the same values in "received" and "size". @dschissler how you've deal with this?

[Ramel]

I seem's I have a problem when I define the "upload_progress_header" variable in the nginx.conf upload location. So the module is working well but I have a problem with this variable. I'll do more tests…