Open draskolnikova opened 7 years ago
You cannot use the directives such assmall_light
and proxy_pass
and proxy_cache_xxx
in same virtual host. Divide them into different virtual servers.
@cubicdaiya divide them into different virtual servers? I still didn't get it.
Show your configuration example for reproducing.
@cubicdaiya there's some NDA on my configuration, can we talk using private communication (eg. email).
can we talk using private communication (eg. email).
No.
I tried with the configuration below. But It seems to work fine in my environment.
server {
small_light on;
small_light_getparam_mode on;
small_light_pattern_define hoge dw=200,dh=200;
location / {
proxy_pass http://example.com;
}
}
If you cannot provide the real configuration, please provide the minimal example configuraiton for reproducing.
@cubicdaiya here is the configuration, i remove FQDN from external resource.
upstream awsdist {
server ip.ad.dr.es:80;
keepalive 128;
}
map $http_user_agent $img_mode {
$default default;
~*chrome of=webp&q=70;
~*opera of=webp&q=70;
}
map $http_user_agent $pattern_mode {
$default default;
~*chrome p=optimized;
~*opera p=optimized;
}
server {
listen 80;
server_name example.com;
access_log /var/log/nginx/example.com-access.log;
error_log /var/log/nginx/example.com-error.log;
location = /favicon.ico {
return 204;
access_log off;
log_not_found off;
location = /robots.txt {
return 204;
access_log off;
log_not_found off;
}
small_light_pattern_define optimized of=webp,q=70;
small_light_buffer 16M;
small_light on;
small_light_getparam_mode on;
location ~ \.(jpe?g|png|gif|webp)$ {
proxy_pass http://awsdist;
proxy_cache_use_stale error timeout updating http_500 http_503 http_504;
proxy_cache_valid 200 302 301 24h;
proxy_cache images;
proxy_cache_valid any 3s;
proxy_cache_lock on;
proxy_cache_revalidate on;
proxy_cache_min_uses 1;
proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=images:512m inactive=60m use_temp_path=off max_size=512m;
proxy_cache_key "$host$uri$is_args$args";
proxy_ignore_headers Cache-Control Expires;
proxy_hide_header X-Cache;
proxy_hide_header Via;
proxy_hide_header ETag;
# this directive, works
#rewrite (?<capt>.*) $capt?$args$img_mode break;
# this directive, not works
rewrite (?<capt>.*) $capt?$args$pattern_mode break;
}
}
If I use rewrite (?<capt>.*) $capt?$args$pattern_mode break;
the images size didn't optimized, but I use rewrite (?<capt>.*) $capt?$args$img_mode break;
, it's works. So the URL would be http://example.com/images.jpg (didn't need add additional parameter(s) after *.jpg)
Update:
Change from 80
to 70
on pattern_define.
Why don't you decrease quality? ( Maybe 80 is too high for WebP.)
@cubicdaiya my bad, the value on pattern_define should be 70
. I was test it using $pattern_mode
, nothing was change. Don't know is it bug or miss configuration, any hints?
The cause is the value of q is too large? In my environment, the converted-webp-image size is larger than original-image when q is large. (e.g. q=80)
Maybe converted-webp-image is smaller than original if q is more low or eliminating q in parameters.
@cubicdaiya
As your request, I remove parameter q
on rewrite rules, here's the result and use $img_mode
directive.
Original Upstream:
accept-ranges:bytes
age:4
cache-control:max-age=604800, public
content-length:68563
content-type:image/jpeg
date:Sun, 23 Oct 2016 11:39:17 GMT
etag:"[removed]"
last-modified:Sat, 25 Jun 2016 23:06:35 GMT
server:AmazonS3
status:200
via:1.1 [removed].cloudfront.net (CloudFront)
x-amz-cf-id:[removed]
x-cache:Hit from cloudfront
Result, I got 67,4KB
on chrome
Proxied images:
cache-control:max-age=604800, public
content-encoding:gzip
content-type:image/webp
date:Sun, 23 Oct 2016 11:39:23 GMT
last-modified:Sat, 25 Jun 2016 23:06:35 GMT
server:AmazonS3
status:200
vary:Accept-Encoding
x-amz-cf-id:HIT from id02
On chrome, I got 66,4KB
.
Tried add proxy_pass_header Content-Length;
on configuration, but no luck :D
content-encoding:gzip
Your WebP image looks gzip-compressed. As a image such as WebP and JPEG is already compressed, they should not be compressed by gzip. ( In most case, image size will be expanded. )
@cubicdaiya
Ok, I was set webp are not compressed by gzip.
Upstream Images: 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
accept-ranges:bytes
age:3
cache-control:max-age=604800, public
content-length:592651
content-type:image/jpeg
date:Sun, 23 Oct 2016 13:08:53 GMT
etag:"[removed]"
last-modified:Thu, 10 Dec 2015 02:40:24 GMT
server:AmazonS3
status:200
via:1.1 [removed].cloudfront.net (CloudFront)
x-amz-cf-id:[removed]
x-cache:Hit from cloudfront
Proxied Images: 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
cache-control:max-age=604800, public
content-length:595062
content-type:image/webp
date:Sun, 23 Oct 2016 13:09:01 GMT
last-modified:Thu, 10 Dec 2015 02:40:24 GMT
server:AmazonS3
status:200
x-amz-cf-id:HIT from id01
Why proxied image file larger than upstream image file? Still using $img_mode
parameter without q
. If you need the file, you can go to this link http://103.52.2.19/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg and check the header, this is the original file before converted to webp.
FYI, this is my system environment.
nginx version: nginx/1.11.4
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
built with OpenSSL 1.0.2j 26 Sep 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-openssl=openssl-1.0.2j --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_stub_status_module --with-http_auth_request_module --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --with-http_xslt_module=dynamic --add-dynamic-module=ngx_cache_purge --add-dynamic-module=nginx-module-vts --add-dynamic-module=headers-more-nginx-module --add-dynamic-module=ngx_small_light --add-dynamic-module=ngx_brotli --add-module=nginx_upstream_check_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_slice_module --with-mail=dynamic --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -Ofast -march=native -ffast-math -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2'
Linux redirector-engineer.dist01-jkt3d.xtremenitro.org 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ImageMagick-6.7.8.9-15.el7_2.x86_64
Version: ImageMagick 6.7.8-9 2016-06-16 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP
Why don't you try to convert your image by convert
?
$ convert -format webp image.jpg image.webp
@cubicdaiya Done, convert it to test the size is lower than original images?
[root@phantom html]# ls -lah 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.*
-rw-r--r--. 1 root root 582K Dec 10 2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root 33K Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
In my envinronment, convert
without quality assignment inflates image. But image is shrinked when quality is 50.
$ convert -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
$ convert -quality 50 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
$ ls -l | grep 0_0
-rw-r--r-- 1 bokko staff 595062 Dec 10 2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r-- 1 bokko staff 674310 Oct 24 00:17 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
-rw-r--r-- 1 bokko staff 26386 Oct 24 00:17 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
Why proxied image file larger than upstream image file?
This phenomenon is not reproduced at all. So I have no idea. Is your image quality just high?
@cubicdaiya I can give you a sandbox and replicate my existing environment. So you can login and reproduce there. I thought the configuration above is same with existing environment, the difference is the FQDN upstream.
How about that?
FYI, I just do and replicate the file like yours, here is the result.
[root@phantom html]# convert -quality 50 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
[root@phantom html]# ls -l | grep 0_0
-rw-r--r--. 1 root root 595062 Dec 10 2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root 33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
-rw-r--r--. 1 root root 33698 Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
[root@phantom html]# convert -quality 10 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp
[root@phantom html]# !ls
ls -l | grep 0_0
-rw-r--r--. 1 root root 595062 Dec 10 2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root 33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp
-rw-r--r--. 1 root root 33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
-rw-r--r--. 1 root root 33698 Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
You can access it through http://103.52.2.19/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp.
Dunno how to reproduce and start from where :D
I'm sorry, but I'm not going so far as to investigate this problem.
@cubicdaiya it's okey bro :) I just give feedback to your hobby then. :+1:
Hello!
Got curious, when I deploy
small_light
module combine withproxy_pass
directive, my goal is to optimize images from external resource. My nginx + small_light + proxy_pass configuration looks like :I tried to check external resource file size using cURL:
external resource
Proxied using nginx + small_light + proxy_pass
Nginx + small_light
The images self-hosted on it's server, not proxied to upstream.
My question is, the size of file didn't optimized when proxied to upstream, see
content-length
's overhead on external resource (592651
bytes) with nginx + small_light + proxy_pass (595062
bytes). We got ~3KBytes overhead ifsmall_light
active on proxy_pass directive.Then, we got only
33698
bytes on non-proxied images.Is it normal?
Update: Bugs appeared when using pattern define, but when I directly invoke using
of=webp&q=80
, it's works.