IPRO not work properly for images optimisation, instead get "extended_cache_links"

[asshaposhnikov]

Hello, i have multiple virtual hosts Nginx server. And turned on IPRO for images, to convert them in webp. But lost a day to get original links to optimised images, but PS give me extended cache .pagespeed. links.

Nginx configuration architecture:

user www-data; worker_processes 1;

error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid;

events { worker_connections 1024; multi_accept on; accept_mutex off; }

http { ...

Pagespeed common settings for all sites

pagespeed FileCachePath /var/www/ram/ngx_pagespeed_cache;
pagespeed FileCacheSizeKb 302400;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed CreateSharedMemoryMetadataCache "/var/www/ram/ngx_pagespeed_cache/" 102400;
pagespeed EnableCachePurge on;

pagespeed ProcessScriptVariables on;
pagespeed MemcachedServers "localhost:11211";

#Pagespeed statistic and admin panel settings
pagespeed Statistics on;
pagespeed StatisticsLogging on;
pagespeed LogDir /var/log/nginx/pagespeed;
pagespeed UsePerVhostStatistics on;

pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;

pagespeed StatisticsDomains Disallow *;
pagespeed GlobalStatisticsDomains Disallow *;
pagespeed MessagesDomains Disallow *;
pagespeed ConsoleDomains Disallow *;
pagespeed AdminDomains Disallow *;
pagespeed GlobalAdminDomains Disallow *;

pagespeed StatisticsDomains Allow system.my-domain.com;
pagespeed GlobalStatisticsDomains Allow system.my-domain.com;
pagespeed MessagesDomains Allow system.my-domain.com;
pagespeed ConsoleDomains Allow system.my-domain.com;
pagespeed AdminDomains Allow system.my-domain.com;
pagespeed GlobalAdminDomains Allow system.my-domain.com;

pagespeed MessageBufferSize 1000000;
pagespeed ImageMaxRewritesAtOnce 10;

Virtual host configuration file:

Detect "webp" image format support

map $http_accept $webp_support {
    default "0";
    "~*image/webp~*" "1";
}

proxy_cache_path /var/www/ram/landing_proxy_cache levels=1:2 keys_zone=lpage:32m max_size=100m inactive=31d use_temp_path=off;

server { listen 1.1.1.1:443 ssl http2; ... pagespeed off; ...

#Serving optimized by pagespeed resources from pagespeed cache(memcached). It give memory economy, because resorces not double in proxy cache. 
if ($uri ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
    set $bypass_cache "1";
    set $dont_store_response_inproxy "1";
}

if ($uri ~* \.(gif|jpg|jpeg|png|webp)$) {
    set $bypass_cache "1";
    set $dont_store_response_inproxy "1";
}

   ...

location / {
    proxy_pass http://pagespeed_isolated;
    proxy_pass_request_headers on;
    proxy_redirect off;

    proxy_cache_key webp=$webp_support$scheme://$http_host$request_uri;
    proxy_cache lpage;

        proxy_cache_valid       200 301 302 31d;

    proxy_cache_revalidate on;
    add_header X-Cache-Status $upstream_cache_status;

    #This parametrs must be. For properly work Lua progs
    proxy_pass_header date;
    proxy_ignore_headers Set-Cookie;
    proxy_ignore_headers Cache-Control;

    proxy_hide_header Cache-Control;
        add_header Cache-Control $new_cache_control_header_val;
    proxy_cache_bypass $bypass_cache;
        proxy_hide_header PS-ShouldBeacon;
        proxy_set_header PS-ShouldBeacon $should_beacon_header_val;

    #Disable double Vary header
    proxy_hide_header Vary;

    #Dont store in cache alredy cached by pagespeed resources - *.pagespeed.* and 
    #images(in case IPRO images will send from pagespeed instead proxy cache)
    proxy_no_cache $dont_store_response_inproxy;

    proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header HTTPS "on";
            proxy_set_header Ssl-Offloaded "1";
            proxy_set_header X-Forwarded-Proto $scheme;

    #This header force pagespeed not make default gzip content compress
    proxy_set_header Accept-Encoding "";

    proxy_set_header Host $host;

    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_background_update off;
        proxy_cache_lock on;
}

}

Pagespeed isolated server, for male optimisation

server { listen localhost:8080; ...

Common pagespeed settings for all sites

include /etc/nginx/pushificator-pagespeed-conf/pushificator-pagespeed.conf;

Content of include file

pagespeed on;

pagespeed HttpCacheCompressionLevel 0;
pagespeed NoTransformOptimizedImages on;
pagespeed PreserveUrlRelativity on;
pagespeed EnableFilters remove_comments;
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters make_google_analytics_async;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters local_storage_cache;

#proxyfied settings
pagespeed RespectXForwardedProto on;

pagespeed ImagePreserveURLs on;

pagespeed AllowVaryOn Accept;

pagespeed DisableFilters extend_cache_images;
pagespeed EnableFilters inline_images;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters convert_to_webp_lossless;

pagespeed EnableFilters convert_to_webp_animated;
pagespeed WebpQualityForSaveData 100;
pagespeed WebpRecompressionQualityForSmallScreens 100;
pagespeed InPlaceResourceOptimization on;
pagespeed EnableFilters in_place_optimize_for_browser;

pagespeed EnableFilters debug;

#to decrease critical path. For remove 1-RTT at fonts load we inline small *.css files, in *.html page
# fonts will be requested after html load. Without waiting for fonts.css load.
#Requirement - css files that loads fonts - must be smaller then "CssInlineMaxBytes" directive.
pagespeed DisableFilters combine_css;
pagespeed EnableFilters inline_css;
pagespeed CssInlineMaxBytes 10240;
    #end of settings file

#Special for this virtual host settings

pagespeed FetchHttps enable;
pagespeed Domain http://localhost;
pagespeed Domain https://my-lp-domain.com;

pagespeed MapOriginDomain "http://localhost" "https://my-lp-domain.com";
pagespeed LoadFromFile "https://my-lp-domain.com" "/var/www/my-lp-domain.com/public_html/";

pagespeed DownstreamCachePurgeLocationPrefix http://localhost:8081/cache_purge/;
pagespeed DownstreamCachePurgeMethod GET;
pagespeed DownstreamCacheRebeaconingKey "123456789";
pagespeed DownstreamCacheRewrittenPercentageThreshold 100;

pagespeed ImplicitCacheTtlMs 2678400000;

pagespeed EnableFilters combine_javascript;
pagespeed MaxCombinedJsBytes 200000;

location ~ "^/ngx_pagespeed_static/" { }

location ~ "^/ngx_pagespeed_beacon$" { }

location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
    add_header "" "";
}

location ~ \.php$ {
    fastcgi_index   index.php;
    fastcgi_intercept_errors on;
    fastcgi_pass unix:/run/php/php7.3-fpm-landingspeed.sock;
    include fastcgi_params;
    fastcgi_param       SCRIPT_FILENAME  $document_root$fastcgi_script_name;
}

location / {
    try_files $uri $uri/ =404;
}

location ~* ^.+\.(webp|jpg|png|ico|gif)$ {
    expires max;
    log_not_found off;
}

location ~* ^.+\.(css|js)$ {
    expires max;
    log_not_found off;
}

}

End Nginx configuration.

Why i have this configuration, and what purposes i have? All request go throw Nginx proxy to pagespeed server. Nginx proxy configured to store in cache only "text/html" content, and purging them by Lua script, if pagespeed server not complete optimisation at 100%, or if CMS backend send purging request for URL. All other files(jpeg/webp/css/js) are send from pagespeed cache (memcached), and it must be fast scheme, i think.

I want to serve images request in IPRO style, but cant get this. Instead i have "extended_cache" style URL. Can you help me solve this? Or it will be more easy to write Lua prog and convert jpg->webp by hands or backend script?

My server - Ubuntu 16 Nginx - compiled from sources. Options: nginx version: nginx/1.15.12 built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11) built with OpenSSL 1.1.1b 26 Feb 2019 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -DNGX_HTTP_HEADERS' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' --add-module=/root/nginx-compile-from-source/ngx_pagespeed/incubator-pagespeed-ngx-1.13.35.2-stable --add-module=/root/nginx-compile-from-source/memc-nginx-module --add-module=/root/nginx-compile-from-source/srcache-nginx-module --add-module=/root/nginx-compile-from-source/ngx_devel_kit_module/ngx_devel_kit --add-module=/root/nginx-compile-from-source/set-misc-nginx-module/set-misc-nginx-module --add-module=/root/nginx-compile-from-source/headers-more-nginx-module/headers-more-nginx-module --add-module=/root/nginx-compile-from-source/replace-filter-nginx-module --with-ld-opt=-Wl,-rpath,/usr/local/lib --add-module=/root/nginx-compile-from-source/ngx_brotli --add-module=/root/nginx-compile-from-source/ngx_cache_purge --add-module=/root/nginx-compile-from-source/lua-nginx-module --add-module=/root/nginx-compile-from-source/ngx_http_geoip2_module

[asshaposhnikov]

Try to turn off all virtual hosts in nginx, except one with pagespeed. Same result - without images IPRO . Images lazy load filter - not work. In version 1.15.8 few month ago - lazy load images work fine. May be 1.15.10 and pagespeed bug? Try Nginx 1.17.0 - not solve.

ipro_not_in_cache	175	1
ipro_not_rewritable	0	0
ipro_recorder_dropped_due_to_load	0	0
ipro_recorder_dropped_due_to_size	0	0
ipro_recorder_failed	175	1
ipro_recorder_inserted_into_cache	0	0
ipro_recorder_not_cacheable	175	1
ipro_recorder_resources	175	1
ipro_served	0	0

[asshaposhnikov]

At first page load, when not optimised at 100%. Some times i can see both - .pagespeed. and IPRO links, with jpeg converted in webp (but without "x-page-speed" header). Then, after page refresh few times, - all images links transform to .pagespeed.

[oschaaf]

I'm not sure I accurately understand the problem, but perhaps you are looking for OptimizeForBandwidth mode?: https://www.modpagespeed.com/doc/optimize-for-bandwidth

[asshaposhnikov]

I think it's in the intricacies of the settings. Namely, in the combination of headers User-agent, Accept. And the pagespeed options: AllowVaryOn ..., InPlaceResourceOptimization, in_place_optimise_for_browser, recompress_images, convert_jpeg_to_webp, convert_to_webp_lossless. Try now you way of decission and headers/options manipulate. Response result here later.

[asshaposhnikov]

I'm not sure I accurately understand the problem, but perhaps you are looking for OptimizeForBandwidth mode?: https://www.modpagespeed.com/doc/optimize-for-bandwidth

Not help. Check both - OptimiseForBandwidth and CoreFilters. Directives aplied in top level (http section) nginx conf. Images links same - .pagespeed.

Example of two optimisation types in same time on picture (IPRO and extended cache).

I try same settings on another server, without nginx proxy. All work fine.

[asshaposhnikov]

Just reboot server much times during another virtual host configure. Check proxied site. All work fine, images in IPRO style. No configuration Nginx/pagespeed changes made. What can I say? Pagespeed - configure itself, then when he considers it necessary)

[jmarantz]

Hi... It doesn't make sense to configure both OptimizeForBandwidth and CoreFilters. You can only set RewriteLevel to one of those.

I think you just want OptimizeForBandwidth.

[asshaposhnikov]

Hi... It doesn't make sense to configure both OptimizeForBandwidth and CoreFilters. You can only set RewriteLevel to one of those.

I think you just want OptimizeForBandwidth.

Hi, ok. Sometimes you want to understand the algorithm of pagespeed, to be sure of the settings made. At one time I thought the reason was in an instrument written by me on Lua. The program, which automatically sends all assets from the head tag, via the push server. https://speed-optimized.wordpress.atlanta18.ru/ , but now all work fine with Core filters and some custome pagespeed settings.