Lofesa
commented
2 years ago No, don´t do that. The rules I have posted don´t work like these you have. Whit your pagespeed Disallow .*; disables all rewites. and then no furter process is done.
Open AnonDev1312 opened 5 years ago
Lofesa
commented
2 years ago No, don´t do that. The rules I have posted don´t work like these you have. Whit your pagespeed Disallow .*; disables all rewites. and then no furter process is done.
ioweb-gr
commented
2 years ago Oh Ok I see, however it does rewrite with the commands above... So many weird behaviors
I reverted to disallowing each extension separately
ioweb-gr
commented
2 years ago Again the typical 4xx status code
Incidentally I managed to find the access_log off commands in nginx and reverted them to access_log on
But I still don't see any serf in the proxy_access_ssl_log
The above url is 200 in the logs
Lofesa
commented
2 years ago Now I can see mayority of files writed, unless svg, gif and some jpg/png. Others image files, like gallery.png is not rewrited becase is loaded by a css file that you have excluded. Some others are loaded by js files, so are out of pagespeed.
In the stats page (pagespeed_global_admin/statistics) have you too many evictions (file_cache_evictions)?
And.. Are you sure all image files are in the nginx location? If you think they are here, for testing purposes, why don´t do the proxy thing?
location ~ ^/(.*\.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|eot|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|svg|swf|tar|tgz|ttf|txt|wav|woff|woff2|xls|xlsx|zip|webp))$ {
try_files $uri $uri/ =404;
}I have added the location directive already. But the thing is I can see like 20 .jpg images rewrited, but another 30 not rewrited.
Messages
jmarantz
commented
2 years ago Sorry I haven't been following all the details but I can tell you that the "Allow" and "Disallow" commands work with wildcards, not regexes. So I don't think any of those commands will work like you think they should :)
ioweb-gr
commented
2 years ago Yes I read the docs and adjusted thanks for the input now they are like this
pagespeed Disallow "*.svg";
pagespeed Disallow "*.woff2";
pagespeed Disallow "*.woff";
pagespeed Disallow "*.ttf";
pagespeed Disallow "*.ico";
pagespeed Disallow "*.js";
pagespeed Disallow "*.css";
pagespeed Disallow "*pub/static/*.html";After reducing the clutter I notice this error message also
Too busy to rewrite image.
What does this mean, how do I fix it?
jmarantz
commented
2 years ago That's not necessarily a problem. It's an "Info" message not an Error. Image optimizations are cached, and when the cache is cold and there are a lot of concurrent requests to get optimized versions of them, the system drops some of them to avoid exceeding an internal queue size.
When the image cannot be delivered optimized it is delivered in its original form. In the meantime, the cache will fill with optimized images and that message should not constantly recur.
ioweb-gr
commented
2 years ago I see, it's beginning to fill more and more images. Almost all jpgs and jpegs have converted now. Some pngs refuse to convert even though they're rewrited. They still serve as png saying they don't need resizing.
I wonder why always in each request a few jpgs wont' convert randomly.
I think that the configuration is working now.
Is it possible that when I didn't disallow every extension specifically the amount of requests was so big that pagespeed couldnt' handle it because it was busy ?
ioweb-gr
commented
2 years ago Then I visit another category page of the website, and the 4xx status code errors come back up again for all images
I have no idea what is going on honestly. One page rewrites, one doesn't
Lofesa
commented
2 years ago Not importat. Pagespeed tries to optimize 8 images at once and take 10ms. to do it, is there are more images or take more than 10ms to do the work, then it do a background fecht so the next hit it get rewrited.
You can use:
pagespeed ImageMaxRewritesAtOnce -1; # This tries as much images the server can do
pagespeed RewriteDeadlinePerFlushMs 5; # This reduce the 10ms to 5.If you are using the LoadFromFIle you must set this:
pagespeed LoadFromFileCacheTtlMs 604800000; #for a week
as far as loaded from disk files dont have any http header, this set the time the optimized resource lives in the pagespeed cache
P.D. the page where I´m trying is https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html?PageSpeedFilters=+debug
In this page I got 16 images not rewrited. From these only 4 are the issue, because the others are in css files, loaded by javascript, are svg or gif, or are from others domains.
the 4 images are:
/pricealert.jpg
/dc3728-003-phsrh000-1500.jpeg
/wilson_roland_garros_tennis_ball_keychain-wr8401401-1.jpg
/dc3580-010-phsfm001-1500.jpgCan you take a look in the message history whats hapen with these images?
ioweb-gr
commented
2 years ago here's one example
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson_roland_garros_tennis_ball_keychain-wr8401401-1.jpg fragment=example.com: remembering recent failure for 298 seconds.
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson_roland_garros_tennis_ball_keychain-wr8401401-1.jpg fragment=example.com: remembering recent failure for 298 seconds.
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson_roland_garros_tennis_ball_keychain-wr8401401-1.jpg
[Mon, 15 Nov 2021 19:33:36 GMT] [Info] [507507] Cache entry is expired: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson_roland_garros_tennis_ball_keychain-wr8401401-1.jpg (fragment=example.com)
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/d/c/dc3580-010-phsfm001-1500.jpg fragment=example.com: remembering recent failure for 298 seconds.
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/d/c/dc3580-010-phsfm001-1500.jpg fragment=example.com: remembering recent failure for 298 seconds.
[Mon, 15 Nov 2021 19:33:37 GMT] [Info] [507507] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/d/c/dc3580-010-phsfm001-1500.jpg
[Mon, 15 Nov 2021 19:33:36 GMT] [Info] [507507] Cache entry is expired: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/d/c/dc3580-010-phsfm001-1500.jpg (fragment=example.com)
``I see, it's beginning to fill more and more images. Almost all jpgs and jpegs have converted now. Some pngs refuse to convert even though they're rewrited. They still serve as png saying they don't need resizing.
Maybe they are transparent images? Anyway, if pagespeed can´t see any image reduction wetbeen the png and the webp, it choses to no convert it.
I wonder why always in each request a few jpgs wont' convert randomly.
I think that the configuration is working now.
Is it possible that when I didn't disallow every extension specifically the amount of requests was so big that pagespeed couldnt' handle it because it was
busy?
See my comment before.
ioweb-gr
commented
2 years ago @Lofesa Would you mind sanitizing this message url that I missed for privacy reasons? https://github.com/apache/incubator-pagespeed-mod/issues/1897#issuecomment-968895233
Also since you got access to the env now would you mind also taking a quick view because this page won't rewrite at all the jpgs /storeviewgreek/tennis-shoes.html?PageSpeedFilters=+debug
It's the same setup so I don't understand why it fails
I added the above config for testing
ioweb-gr
commented
2 years ago What's really puzzling me is that it's doing a shrink in this file
[Mon, 15 Nov 2021 19:42:30 GMT] [Info] [519193] https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html:11170-11174: Shrinking image `https://subdomain.example.com/pub/media/catalog/product/cache/f5f3da80ad7b670245aea7e970662954/w/r/wr8413601.jpg' (29431 bytes) to `https://subdomain.example.com/pub/media/catalog/product/cache/f5f3da80ad7b670245aea7e970662954/w/r/xwr8413601.jpg.pagespeed.ic.75CddY7jwy.webp' (13942 bytes)
And yet it serves it as jpg instead of webp.
jmarantz
commented
2 years ago RE "serves as jpg instead of web": in what context?
Accept header?Lofesa
commented
2 years ago @jmarantz Files in the same folder, 1 get rewrited and serve webp version but others not, the file are in place, it can show with a ll bash command, but in the debug comments get the <--- 404'errors prevent image rewrite.
Some others images are optimized by IPRO, but stay in this transient state for ever, and is in the html code.
My bet is here are a cache in front of the server and this cache serves unoptimized resources, optimized by IPRO or fully optmized, but @ioweb-gr says there is no cache in front of the server.
The main request have these headers x-magento-cache-control: and x-magento-cache-debug and all the info I get on these headers are about varnish, that make sense on whats is happening here.
ioweb-gr
commented
2 years ago Well these headers are just debug info on whether you're hitting the application Full Page Cache for the response for which the backend can be a anything. Most of the time it's varnish but we chose redis for this instance. Therefore it's not varnish that's doing this.
Also nginx cache is not utilized. It's disabled so I'm not sure what could be caching other than that.
We don't have cloudflare or any CDN either.
Lofesa
commented
2 years ago @ioweb-gr
Can you take a look in /etc and search for some-file.vlc and grep if there is some like this unset resp.http.X-Varnish;
This hide the varnish header.
Anyway, can you disable any cache in the CMS?
https://help.nexcess.net/magento/how-to-disable-caching-in-magento
ioweb-gr
commented
2 years ago $ find /etc/ -name "*.vlc" | wc -l
0Disabled full page cache and block cache that magento utilizes which affect the frontend. Not turning internal framework caches off because the page pretty much won't load.
Lofesa
commented
2 years ago With pagespeed disabled (?PageSpeed=unplugged) I can see images w/o no cache-control: max-age like /pricealert.jpg (this image have cache-control: public ) Others have cache-control: max-age=1209600, s-maxage=10 like /h006-a_1_1.jpg...
Can you make all images have only cache-control: max-age=1209600 (or what time you will but greater than 0) and no other header?
ioweb-gr
commented
2 years ago To be honest I'm at a loss here as I'm not sure how to do that.
Will it do the trick if in the location block for the static files I temporarily add a
proxy_hide_header
and an
add_header for the cache-control?
ioweb-gr
commented
2 years ago For now I've added using the add_header for the cache control and it seems to show ok
I also added a header to the location directive for static files to make sure it's processed by nginx and not by apache.
I added a similar header in apache requests to differentiate to requests processed by apache and nginx
Should I add etag off too?
Expires directive is gone.
Of course the 4xx error is still there.
Lofesa
commented
2 years ago If you have set the location as stated in this comment https://github.com/apache/incubator-pagespeed-mod/issues/1897#issuecomment-969181614, no proxy headers are in place. So only add_header is needed add_header "control-cache" "max-age=86400"; # 1 day for testing. If a proxy is involved, then yes, you have to set
I read in the CMS doc about how to set cache headers, but all I have found is related to html pages, not for statics resource. Only in ths page https://devdocs.magento.com/guides/v2.4/config-guide/cache/static-content-signing.html talk about statics files and a expires headers. But cache-control header have preference over expires.
ioweb-gr
commented
2 years ago Yes exactly, Magento only sets headers for the application dynamic content.
For static files it's a combination of rewrites plus .htaccess rules
For now I've added the cache control header as suggested but I didn't see any improvement.
Lofesa
commented
2 years ago I
For now I've added using the add_header for the cache control and it seems to show ok
I also added a header to the location directive for static files to make sure it's processed by nginx and not by apache.
I added a similar header in apache requests to differentiate to requests processed by apache and nginx
Should I add etag off too?
Expires directive is gone.
Of course the 4xx error is still there.
I still viewing mixed cache-control header and ioweb-served-by Only tested images, some have it, some apache and some nginx, and other don´t have the ioweb-served-by header. and 2 or 3 images returns 404. All this with pagespeed disabled ( url?PageSpeed=unplugged)
All files on https://subdomain.example.com/pub/static comes from apache
700040-babolat_all.jpg
dog_dampener.jpg
logo-etennis.icoreturns 404
More strange, image files served with ?PageSpeed=unplugged are not the same as served with ?PageSpeedFilters=+debug Whit unplugged I have these files with 404, with debug, no. Like a page cache with differen code for each url parameter. Whit unplugged I get files with etag headers from IPRO, but this no posible, pagespeed is disabled..
Image +debug
Image unplugged
Don´t have ioweb-served-by
Image served by apache, so proxy is in place and file is not under nginx location
Image served unplugged (pagespeed totally off) but have haders from IPRO (the etag header)
This make me thinking images are cached somewhere and served with the headers they have been stored.
Lofesa
commented
2 years ago Yes exactly, Magento only sets headers for the application dynamic content.
For static files it's a combination of rewrites plus .htaccess rules
For now I've added the cache control header as suggested but I didn't see any improvement.
For example the image /pricealert.jpg some times have cache-control: public and don´t be rewrited, but some others times it got cache-control: public, s-maxage=10 and got rewrited....
borishinzer
commented
2 years ago We had the same issue, which was basically caused by our deployment (using deployeer with multiple symlinks to product/current) and making it impossible to follow the pagespeed module to get the files from the right folder structure.
We resolved it by adding the LoadFromFile like this to the nginx settings of the subdomain: pagespeed LoadFromFile "https://stage.domain.de/fileadmin/" "/var/www/vhosts/domain.de/stage/public/fileadmin/";
ioweb-gr
commented
2 years ago @borishinzer Unfortunately this doesn't work for us due to versioning. The URL changes every time we deploy so the file is not found with LoadFromFile on production deployment.
borishinzer
commented
2 years ago @ioweb-gr , yes we do the same or similar. See our structure (also taken from a plesk site):
lrwxrwxrwx 1 user psacln 25 Dec 15 2020 httpdocs -> sites/production/current/
lrwxrwxrwx 1 user psacln 20 Jan 12 2021 stage -> sites/stage/current/Inside folder sites:
drwxr-xr-x 5 user psacln 4096 Dec 6 11:09 production
drwxr-xr-x 5 user psacln 4096 Dec 6 16:21 stageInside folder production:
drwxr-xr-x 2 vdvonlin psacln 4096 Dec 6 11:09 .dep
lrwxrwxrwx 1 vdvonlin psacln 11 Dec 6 11:09 current -> releases/48
drwxr-xr-x 7 vdvonlin psacln 4096 Dec 6 11:09 releases
drwxr-xr-x 4 vdvonlin psacln 4096 Nov 28 02:15 sharedCurrent is always pointing to latest code release files (always at a different location).
The folder shared/public contains the assets (like images, pdf, etc), making it possible to access them with the LoadFromFile even after deployments.
Mod PageSpeed debug is showing a lot of errors like these:
I tried this but it didn't work:
Any help would be appreciated