search

apache / incubator-pagespeed-ngx

Automatic PageSpeed optimization module for Nginx
http://ngxpagespeed.com/
Apache License 2.0
4.37k stars 365 forks source link

How to properly configure nginx proxy_cache as ngx_pagespeed downstream cache? #1397

Open kaixulucky opened 7 years ago

kaixulucky commented 7 years ago

I am a novice, please help me, very grateful！ In my test environment The following log is the result of completely clearing the proxy_cache and ngx_pagespeed caches 10.0.0.19 is the server IP, 172.31.0.253 is the client, each request to use ctrl + f5, how do not feel like https://modpagespeed.com/doc/downstream-caching introduced the kind, The user --- proxy_cache --- ngx_pagespeed, now becomes the user ---- ngx_pagespeed ---- proxy_cache,What is my place where there is no configuration right?

[root@hjl100019 logs]# tail -f access.log |grep logo First request:10.0.0.19 <> 24/Mar/2017:17:23:59 +0800 <> www.a.com <> GET /Public/images/logo.jpg HTTP/1.1 <> 200 <> 13179 <> - <> 10.0.0.3:80 <> 0.003 <> 0.003 <> - <> MISS <> Serf/1.3.8 (mod_pagespeed/1.12.34.2-0) <> - Second request:172.31.0.253 <> 24/Mar/2017:17:23:59 +0800 <> www.a.com <> GET /Public/images/xlogo.jpg.pagespeed.ic.f524Si0mai.jpg HTTP/1.1 <> 200 <> 5180 <> - <> - <> - <> 0.019 <> - <> - <> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 <> kb153035nivkjgmlgr1dgtl7a3glr702 Third request:172.31.0.253 <> 24/Mar/2017:17:24:06 +0800 <> www.a.com <> GET /Public/images/xlogo.jpg.pagespeed.ic.f524Si0mai.jpg HTTP/1.1 <> 200 <> 5180 <> http://www.a.com/index.php/pchome/index/index.html <> - <> - <> 0.000 <> - <> - <> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 <> kb153035nivkjgmlgr1dgtl7a3glr702 Fourth request:172.31.0.253 <> 24/Mar/2017:17:25:19 +0800 <> www.a.com <> GET /Public/images/xlogo.jpg.pagespeed.ic.f524Si0mai.jpg HTTP/1.1 <> 200 <> 5180 <> http://www.a.com/index.php/pchome/index/index.html <> - <> - <> 0.001 <> - <> - <> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 <> kb153035nivkjgmlgr1dgtl7a3glr702 Fifth request:172.31.0.253 <> 24/Mar/2017:17:25:33 +0800 <> www.a.com <> GET /Public/images/xlogo.jpg.pagespeed.ic.f524Si0mai.jpg HTTP/1.1 <> 200 <> 5180 <> - <> - <> - <> 0.001 <> - <> - <> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 <> kb153035nivkjgmlgr1dgtl7a3glr702

Request the optimized url [root@hjl100019 pagespeed_cache]# curl --head www.a.com/Public/images/xlogo.jpg.pagespeed.ic.f524Si0mai.jpg HTTP/1.1 200 OK Date: Fri, 24 Mar 2017 09:40:05 GMT Content-Type: image/jpeg Content-Length: 5172 Connection: keep-alive Vary: Accept-Encoding Link: http://www.a.com/Public/images/logo.jpg; rel="canonical" Server: HWS **X-Cache: MISS(nginx) ##But the log inside the field is empty** Accept-Ranges: bytes X-Original-Content-Length: 13179 X-Page-Speed: H_Speed

Request the original url [root@hjl100019 pagespeed_cache]# curl --head www.a.com/Public/images/logo.jpg HTTP/1.1 200 OK Date: Fri, 24 Mar 2017 09:39:54 GMT Content-Type: image/jpeg Content-Length: 13179 Connection: keep-alive Vary: Accept-Encoding Last-Modified: Fri, 03 Jun 2016 03:27:30 GMT ETag: "c21d0-337b-534574d770c80" Server: HWS Expires: Fri, 24 Mar 2017 09:49:54 GMT Cache-Control: max-age=600 X-Cache: HIT(nginx) Accept-Ranges: bytes

# The configuration is as follows:

  1. nginx.conf file contents:

http { ........... Omitted ........... gzip_static on; gzip on; gzip_disable "MSIE[1-6]."; gzip_min_length 1k; gzip_buffers 4 16k; gzip_comp_level 4; gzip_vary on; gzip_proxied any; gzip_types text/plain application/x-javascript application/javascript text/css text/javascript application/x-httpd-php image/jpeg image/gif image/png; proxy_temp_path /home/ngx_cache/temp; proxy_cache_path /home/ngx_cache/cache levels=1:2 keys_zone=file_cache:512m inactive=1d max_size=200g; include conf.d/pagespeed/pagespeed_http.conf; server { listen 80 backlog=65535 reuseport fastopen=256; listen 443 backlog=65535 reuseport fastopen=256 ssl http2; server_name ~^(www|m|oa).a.com$; location / { proxy_pass http://my_webserver_group; include conf.d/proxy.conf; } location ~ /purge(/.) { allow 127.0.0.1; allow 172.16.0.0/12; proxy_cache_purge file_cache $host$1$is_args$args; } Only for static resources to do optimization and cache, does not include html file resources. location ~ ..(js|css|ico|gif|jpg|jpeg|png|bmp|swf)$ { expires 600; proxy_pass http://my_webserver_group; include conf.d/proxy.conf; proxy_cache_valid 200 304 10m; include conf.d/ngx_cache.conf; pagespeed on;
pagespeed DownstreamCachePurgeLocationPrefix "http://www.a.com/purge"; pagespeed DownstreamCachePurgeLocationPrefix "https://www.b.com/purge"; pagespeed LoadFromFileCacheTtlMs 600000; } location /backstatus {check_status;allow 172.16.0.0/12;deny all;} location /stub_status {stub_status on;allow 172.16.0.0/12;deny all;} location ~ ^/opgui.*.php$ { allow 127.0.0.1;allow 172.16.0.0/24; deny all;} location ~ ^/phptest.php$ { allow 172.16.0.0/24; deny all;} location /ngx_pagespeed_statistics { allow 127.0.0.1; allow 172.16.0.0/12; deny all; } location /ngx_pagespeed_global_statistics { allow 127.0.0.1; allow 172.16.0.0/12; deny all; } location /ngx_pagespeed_message { allow 127.0.0.1; allow 172.16.0.0/12; deny all; } location /pagespeed_console { allow 127.0.0.1; allow 172.16.0.0/12; deny all; } location ~ ^/pagespeed_admin { allow 127.0.0.1; allow 172.16.0.0/12; deny all; } location ~ ^/pagespeed_global_admin { allow 127.0.0.1; allow 172.16.0.0/12; deny all; }

location ~ ".pagespeed.([a-z].)?[a-z]{2}.[^.]{10}.[^.]+" { add_header "" ""; } } pagespeed Statistics on; pagespeed StatisticsLogging on; pagespeed LogDir /var/log/pagespeed;

pagespeed StatisticsPath /ngx_pagespeed_statistics; pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics; pagespeed MessagesPath /ngx_pagespeed_message; pagespeed ConsolePath /pagespeed_console; pagespeed AdminPath /pagespeed_admin; pagespeed GlobalAdminPath /pagespeed_global_admin; }

  1. Ngx_cache.conf file contents:

proxy_cache file_cache; proxy_cache_valid 301 302 1h; proxy_cache_valid any 1m; proxy_cache_key $host$uri$is_args$args; proxy_cache_use_stale updating; proxy_cache_revalidate on; proxy_cache_lock on; proxy_cache_lock_timeout 2s;

proxy_cache_min_uses 2;

add_header X-Cache-Status $upstream_cache_status;

add_header X-Cache $upstream_cache_status(nginx); proxy_ignore_headers "Expires" "Set-Cookie" "Cache-Control" "Vary";

  1. pagespeed_http.conf file contents

pagespeed FileCachePath /home/pagespeed_cache; pagespeed DisableRewriteOnNoTransform off; pagespeed LowercaseHtmlNames on; pagespeed ModifyCachingHeaders off; pagespeed ListOutstandingUrlsOnError on; pagespeed PreserveUrlRelativity on; pagespeed XHeaderValue "H_Speed"; pagespeed RewriteLevel CoreFilters;

pagespeed Domain .a.co; pagespeed Domain .b.com; pagespeed Domain https://*.a.co; ##必须要 pagespeed Domain https://*.b.com;#好像可以不要 pagespeed SslCertDirectory /etc/pki/tls/certs;#好像可以不要 pagespeed SslCertFile /etc/pki/tls/cert.pem;#好像可以不要

LRU缓存

pagespeed LRUCacheKbPerProcess 32768; pagespeed LRUCacheByteLimit 65536; pagespeed MessageBufferSize 100000;##如果不设置控制台Message History菜单，显示Writing to mod_pagespeed_message failed. Verify that MessageBufferSize is not set to 0 in pagespeed.conf. pagespeed DefaultSharedMemoryCacheKB 131072; pagespeed CreateSharedMemoryMetadataCache "/home/pagespeed_cache" 131072;

pagespeed EnableCachePurge on; pagespeed PurgeMethod PURGE;

pagespeed UseNativeFetcher on; #疑似启用后不能进行https提取"as https is not supported"

pagespeed FetchHttps enable;

resolver 127.0.0.1;

pagespeed FetcherTimeoutMs 500; pagespeed InPlaceResourceOptimization on; pagespeed EnableFilters in_place_optimize_for_browser;

pagespeed InPlaceSMaxAgeSec 100;

#######20161125 pagespeed ImageMaxRewritesAtOnce 0; pagespeed StatisticsLoggingIntervalMs 60000; pagespeed StatisticsLoggingMaxFileSizeKb 1024; ##############20170225############################

pagespeed imagepreserveurls true;

pagespeed csspreserveurls true;

pagespeed jspreserveurls true;

pagespeed ForceCaching on;

pagespeed ImplicitCacheTtlMs 31536000000;

pagespeed UsePerVhostStatistics on; pagespeed DownstreamCachePurgeMethod GET; pagespeed DownstreamCacheRebeaconingKey "abcd,.1234";

kaixulucky commented 7 years ago

My environment is currently proxy_cache and ngx_pagespeed deployed in the same server and the same port, is this feasible?

oschaaf commented 7 years ago

The downstream cache feature was designed with having a caching proxy in front of the server that runs pagespeed optimization in mind. If you run ngx_pagespeed with a server{} set up as a caching proxy, I think your best bet is caching the unoptimized html. Pagespeed will optimize the html on the fly as nginx serves it from cache, and use its own caches to serve .pagespeed. urls and any in-place optimized versions.

If you want the downstream cache feature to work, and running pagespeed at the webserver is not an option, then I think you would need two proxies (one for caching the html, and one behind that to proxy to the webserver and run ngx_pagespeed).

kaixulucky commented 7 years ago

@oschaaf Please forgive me for this rookie, and I am not good at english, and I am translating Chinese into English through google. I want to only cache and optimize the picture resources, for the "location ~ .. (js | css | ico | gif | jpg | jpeg | png | bmp | swf)" enable pagespeed on, in addition to all off. You say: "then I think you would need two proxies (one for caching the html, and one behind that to proxy to the webserver and run ngx_pagespeed)." Meaning whether for example: "user requests image resources to port 80 (open proxy_cache ) ----- reverse proxy to native port 8081 (enable pagespeed on) ------ reverse proxy to backend server.

jmarantz commented 7 years ago

I don't think you need any proxies for that. Just turn PageSpeed on in your server block, and it will optimize & cache static assets for you -- at least the ones it knows about (js, css, gif, jpg, jpeg, png). You may not need any further caching for those.

You can use some other kind of cache for the file types that PageSpeed doesn't optimize (ico bmp swf).

On Tue, Mar 28, 2017 at 6:11 AM, kaixulucky notifications@github.com wrote:

@oschaaf https://github.com/oschaaf Please forgive me for this rookie, and I am not good at english, and I am translating Chinese into English through google. I want to only cache and optimize the picture resources, for the "location ~ .. (js | css | ico | gif | jpg | jpeg | png | bmp | swf)" enable pagespeed on, in addition to all off. You say: "then I think you would need two proxies (one for caching the html, and one behind that to proxy to the webserver and run ngx_pagespeed)." Meaning whether for example: "user requests image resources to port 80 (open proxy_cache ) ----- reverse proxy to native port 8081 (enable pagespeed on) ------ reverse proxy to backend server.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/1397#issuecomment-289724904, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2kPVeEt5yp-UPBYwI8tzx1bCn641Goks5rqNzogaJpZM4MoB9H .

kaixulucky commented 7 years ago

@jmarantz Hello！ you are saying: "I can not need downstream cache (such as proxy_cache), directly in the desired location pagespeed on, is it? There pagespeed can not support optimization (ico bmp swf) these resources, is it?