search

apache / incubator-pagespeed-ngx

Automatic PageSpeed optimization module for Nginx
http://ngxpagespeed.com/
Apache License 2.0
4.37k stars 365 forks source link

MapProxyDomain OK but 404 #1709

Open ufgeorge opened 4 years ago

ufgeorge commented 4 years ago

I configure pagespeed MapProxyDomain https://$host/shared_img/ https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com;

I see the html source and the url did change to what I want and image can be accessed by changed url. After a few days, there are 404 in nginx log of these changed url.

How did this happen? And how to troubleshoot it? TKX very much.

George

Lofesa commented 4 years ago

Maybe pagespeed cache size is too small and the assets got evicted? See your eviction numbers in stats page...

ufgeorge commented 4 years ago

@Lofesa I only see file_cache_evicons and value is 0.

But I am using memcache. not file cache.

jmarantz commented 4 years ago

Cache eviction should not cause a 404 unless something else is wrong.

Cache expirations also happen.

Is it possible that the origin asset moved?

ufgeorge commented 4 years ago

@jmarantz You can see source from https://course.soundshine.com.tw/?m=Homepage&c=Index&a=index&PageSpeedFilters=+debug

The rewrite url is https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png.pagespeed.ce.06-1T8jikK.png And original img url https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com/256/256F15651848520587.png is accessable.

jmarantz commented 4 years ago

That does not look like the result of MapProxyDomain. Can you share your other domain-related config?

Lofesa commented 4 years ago

Or maybe the image takes too much time to load. The https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com/256/256F15651848520587.png for me takes 25.69s to load

ufgeorge commented 4 years ago

That does not look like the result of MapProxyDomain. Can you share your other domain-related config?

I have several vhost use the same nginx.

pagespeed Domain https://fonts.googleapis.com;

pagespeed MapProxyDomain https://$host/shared_img/ https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com;

pagespeed LoadFromFile "https://$host/" "$document_root/"; pagespeed LoadFromFileRuleMatch disallow .; pagespeed LoadFromFileRuleMatch allow .(ico|pdf|swf|eot|woff|woff2|ttf|otf|css|js|jpeg|jpg|png|gif|svg|svgz|mpg|mpeg|mp3|m4a|m4v|mp4|ogg|wmv|mov|mng|3gpp|3gp|webp|webm|flv|avi|asx|asf)(\?.)?$ps_dollar;

ufgeorge commented 4 years ago

Or maybe the image takes too much time to load. The https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com/256/256F15651848520587.png for me takes 25.69s to load

Not very sure why? If timeout for loading img for rewrite, the img url should not rewrite. I see 404 which means img url rewrite succesful. It means img load and rewrite ok and save into cache?

jmarantz commented 4 years ago

RE "long time to load": that should not cause 404; it should cause PageSpeed to elect not to optimize that resource.

RE "pagespeed Domain https://fonts.googleapis.com;" that is definitely wrong. You can only apply pagespeed Domain to domains that run PageSpeed, and fonts.googleapis.com does not. The existence of that directive in your config file can cause broken pages.

Based on the URLs you gave earlier, I was expecting to see a RewriteDomain rule that was similarly misconfigured. RewriteDomain basically means the two domains reference the same context, and you want to map one to the other. It's quite different from MapProxyDomain which is the only config mechanism provided to optimize assets on another domain.

Are there any other RewriteDomain directives in your config?

Lofesa commented 4 years ago

The image https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png gets a 200 status from alibabacloud. Maybe the rewrited image https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png.pagespeed.ce.06-1T8jikK.png is requested to alibabacloud and it don´t have it?

ufgeorge commented 4 years ago

Are there any other RewriteDomain directives in your config?

There is no any RewriteDomain in my config. The whole pagespeed config as below

PageSpeed configurations

pagespeed on; pagespeed ProcessScriptVariables all; pagespeed FetchWithGzip off;

pagespeed XHeaderValue "Powered By google_pagespeed";

PageSpeed filter settings

pagespeed RewriteLevel CoreFilters; pagespeed RespectVary off; pagespeed DisableRewriteOnNoTransform on; pagespeed NoTransformOptimizedImages on; pagespeed SupportNoScriptEnabled false; pagespeed ModifyCachingHeaders on; pagespeed ListOutstandingUrlsOnError on; pagespeed MaxCacheableContentLength -1; pagespeed DefaultSharedMemoryCacheKB 51200; pagespeed CreateSharedMemoryMetadataCache "/var/cache/ngx_pagespeed/" 51200; pagespeed FileCachePath "/var/cache/ngx_pagespeed/"; pagespeed FileCacheSizeKb 1024000; pagespeed FileCacheCleanIntervalMs 3600000; pagespeed FileCacheInodeLimit 500000; pagespeed LRUCacheKbPerProcess 8192; pagespeed LRUCacheByteLimit 16384; pagespeed HttpCacheCompressionLevel 9; pagespeed InlineResourcesWithoutExplicitAuthorization Script,Stylesheet; pagespeed LazyloadImagesAfterOnload off;

pagespeed LoadFromFileCacheTtlMs 604800000;

pagespeed RewriteDeadlinePerFlushMs 20; pagespeed CacheFragment eletang_st;

pagespeed JsInlineMaxBytes 10480;

Speed up PageSpeed by storing it in the super duper fast memcached

pagespeed MemcachedThreads 1; pagespeed MemcachedServers "x.x.x.x:11211";

PageSpeed admin pages configuration

pagespeed UsePerVhostStatistics off; pagespeed Statistics on; pagespeed StatisticsLogging on; pagespeed StatisticsLoggingIntervalMs 60000; pagespeed StatisticsLoggingMaxFileSizeKb 1024; pagespeed MessageBufferSize 100000; pagespeed LogDir /var/log/pagespeed; pagespeed StatisticsPath /ngx_pagespeed_statistics; pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics; pagespeed MessagesPath /ngx_pagespeed_message; pagespeed ConsolePath /pagespeed_console; pagespeed AdminPath /pagespeed_admin; pagespeed GlobalAdminPath /pagespeed_global_admin;

PageSpeed Cache Purge

pagespeed EnableCachePurge on; pagespeed PurgeMethod PURGE; pagespeed DownstreamCacheRewrittenPercentageThreshold 95;

pagespeed Domain https://fonts.googleapis.com;

pagespeed MapProxyDomain https://$host/doctrans-tw/ https://doctrans-tw.eletang.com.tw;

pagespeed MapProxyDomain https://$host/shared_img/ https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com; pagespeed MapProxyDomain https://$host/source_img/ https://source.eletang.com; pagespeed LoadFromFile "https://$host/" "$document_root/"; pagespeed LoadFromFileRuleMatch disallow .; pagespeed LoadFromFileRuleMatch allow .(ico|pdf|swf|eot|woff|woff2|ttf|otf|css|js|jpeg|jpg|png|gif|svg|svgz|mpg|mpeg|mp3|m4a|m4v|mp4|ogg|wmv|mov|mng|3gpp|3gp|webp|webm|flv|avi|asx|asf)(\?.)?$ps_dollar; pagespeed EnableFilters lazyload_images; pagespeed EnableFilters dedup_inlined_images; pagespeed EnableFilters remove_comments; pagespeed EnableFilters move_css_to_head; pagespeed EnableFilters move_css_above_scripts; pagespeed EnableFilters prioritize_critical_css; pagespeed EnableFilters inline_google_font_css; pagespeed EnableFilters remove_quotes; pagespeed EnableFilters collapse_whitespace; pagespeed EnableFilters in_place_optimize_for_browser;

pagespeed EnableFilters elide_attributes;

pagespeed EnableFilters insert_dns_prefetch; pagespeed EnableFilters hint_preload_subresources;

CoreFilters default

pagespeed EnableFilters combine_javascript;

pagespeed EnableFilters rewrite_javascript;

jmarantz commented 4 years ago

Do you have downstream caching enabled? E.g. a Varnish cache between pagespeed and your clients? If so, can you flush that cache and see if the problem persists?

Maybe there was a bad RewriteDomain or Domain directive in your config, which has since been removed, but a downstream cache is holding onto a broken html page?

On Thu, Aug 27, 2020 at 11:07 PM ufgeorge notifications@github.com wrote:

Are there any other RewriteDomain directives in your config?

There is no any RewriteDomain in my config. The whole pagespeed config as below

`## PageSpeed configurations pagespeed on; pagespeed ProcessScriptVariables all; pagespeed FetchWithGzip off;

pagespeed XHeaderValue "Powered By google_pagespeed";

PageSpeed filter settings

pagespeed RewriteLevel CoreFilters; pagespeed RespectVary off; pagespeed DisableRewriteOnNoTransform on; pagespeed NoTransformOptimizedImages on; pagespeed SupportNoScriptEnabled false; pagespeed ModifyCachingHeaders on; pagespeed ListOutstandingUrlsOnError on; pagespeed MaxCacheableContentLength -1; pagespeed DefaultSharedMemoryCacheKB 51200; pagespeed CreateSharedMemoryMetadataCache "/var/cache/ngx_pagespeed/" 51200; pagespeed FileCachePath "/var/cache/ngx_pagespeed/"; pagespeed FileCacheSizeKb 1024000; pagespeed FileCacheCleanIntervalMs 3600000; pagespeed FileCacheInodeLimit 500000; pagespeed LRUCacheKbPerProcess 8192; pagespeed LRUCacheByteLimit 16384; pagespeed HttpCacheCompressionLevel 9; pagespeed InlineResourcesWithoutExplicitAuthorization Script,Stylesheet; pagespeed LazyloadImagesAfterOnload off;

pagespeed LoadFromFileCacheTtlMs 604800000;

pagespeed RewriteDeadlinePerFlushMs 20; pagespeed CacheFragment eletang_st;

pagespeed JsInlineMaxBytes 10480;

Speed up PageSpeed by storing it in the super duper fast memcached

pagespeed MemcachedThreads 1; pagespeed MemcachedServers "x.x.x.x:11211"; PageSpeed admin pages configuration

pagespeed UsePerVhostStatistics off; pagespeed Statistics on; pagespeed StatisticsLogging on; pagespeed StatisticsLoggingIntervalMs 60000; pagespeed StatisticsLoggingMaxFileSizeKb 1024; pagespeed MessageBufferSize 100000; pagespeed LogDir /var/log/pagespeed; pagespeed StatisticsPath /ngx_pagespeed_statistics; pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics; pagespeed MessagesPath /ngx_pagespeed_message; pagespeed ConsolePath /pagespeed_console; pagespeed AdminPath /pagespeed_admin; pagespeed GlobalAdminPath /pagespeed_global_admin; PageSpeed Cache Purge

pagespeed EnableCachePurge on; pagespeed PurgeMethod PURGE; pagespeed DownstreamCacheRewrittenPercentageThreshold 95;

pagespeed Domain https://fonts.googleapis.com;

pagespeed MapProxyDomain https://$host/doctrans-tw/

https://doctrans-tw.eletang.com.tw; pagespeed MapProxyDomain https://$host/shared_img/ https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com; pagespeed MapProxyDomain https://$host/source_img/ https://source.eletang.com;

pagespeed LoadFromFile "https://$host/" "$document_root/"; pagespeed LoadFromFileRuleMatch disallow . ; pagespeed LoadFromFileRuleMatch allow .(ico|pdf|swf|eot|woff|woff2|ttf|otf|css|js|jpeg|jpg|png|gif|svg|svgz|mpg|mpeg|mp3|m4a|m4v|mp4|ogg|wmv|mov|mng|3gpp|3gp|webp|webm|flv|avi|asx|asf)(?. )?$ps_dollar;

pagespeed EnableFilters lazyload_images; pagespeed EnableFilters dedup_inlined_images;

pagespeed EnableFilters remove_comments; pagespeed EnableFilters move_css_to_head; pagespeed EnableFilters move_css_above_scripts; pagespeed EnableFilters prioritize_critical_css; pagespeed EnableFilters inline_google_font_css;

pagespeed EnableFilters remove_quotes;

pagespeed EnableFilters collapse_whitespace; pagespeed EnableFilters in_place_optimize_for_browser;

pagespeed EnableFilters elide_attributes;

pagespeed EnableFilters insert_dns_prefetch; pagespeed EnableFilters hint_preload_subresources;

CoreFilters default

pagespeed EnableFilters combine_javascript;

pagespeed EnableFilters rewrite_javascript;`

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1709#issuecomment-682299409, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPLBWE453VQHDXBRLL3SC4NOBANCNFSM4QKFZZEQ .

ufgeorge commented 4 years ago

Do you have downstream caching enabled? E.g. a Varnish cache between pagespeed and your clients? If so, can you flush that cache and see if the problem persists? Maybe there was a bad RewriteDomain or Domain directive in your config, which has since been removed, but a downstream cache is holding onto a broken html page?.

No, there is no any cache between client and server hosted this image.

ufgeorge commented 4 years ago

The image https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png gets a 200 status from alibabacloud. Maybe the rewrited image https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png.pagespeed.ce.06-1T8jikK.png is requested to alibabacloud and it don´t have it?

The original img url https://doctrans-tw.oss-cn-shenzhen.aliyuncs.com/256/256F15651848520587.png is ali cloud storage url. So rewrite url https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png still from ali cloud seems no issue.

But 404 url is https://course.soundshine.com.tw/shared_img/256/256F15651848520587.png.pagespeed.ce.06-1T8jikK.png . It looks pagespeed rewrite not only url but also the img itself. Of course ali cloud will not have this img. I assume this rewrited img should be in pagespeed cache but not. So cause 404.

jmarantz commented 4 years ago

Sorry about my confusion -- I think your config looks OK mostly, and there's a possibility of a bug, because the example page for MapProxyDomain is broken also.

The only other suspicion I have is there are a few directives you are using that look suspicious:

pagespeed ModifyCachingHeaders on;
pagespeed RespectVary off;

Setting those things means that a proxy-cache at a users ISP or your CDN can capture PageSpeed's rewritten URL and serve it even if stale, or with user-agent-specific optimizations in it that might not be appropriate for all browsers (e.g. webp transcode).

Could you try removing those configs? In general PageSpeed should work well with minimal configuration and advanced options are available if needed, but you should make sure you understand why you are overriding defaults.

oschaaf commented 4 years ago

Just fixed the online example, that was a config error on the server side.

jmarantz commented 4 years ago

one other thought: maybe ngx_pagespeed is not activated on https://$host/source_img/ and you should map your externally hosted assets onto https://$host/some_directory_with_pagespeed_enabled/source_img?