Open AnonDev1312 opened 5 years ago
Lofesa
commented
5 years ago Hi Have you enabled https fech in pagespeed? I tried to debug with https://www.ni-dieu-ni-maitre.com/?PageSpeedFilters=+debug but get redirected to the main page so can´t view the debug messages.
AnonDev1312
commented
5 years ago I thought https fetch was enabled by default?
HTTPS fetching is built in and is enabled by default as of 1.10.33.0.
If I add this to my htaccess:
ModPagespeedFetchHttps enable,allow_self_signed
I get an Internal Server Error.
I have disabled the redirect using this URL: https://www.ni-dieu-ni-maitre.com/?PageSpeedFilters=+debug&noRedirect=1
AnonDev1312
commented
5 years ago I just noticed that pagespeed started converting some images to webp (not sure why, as I didnt change anything since I posted here)
However, the webp images are returning 404 errors, i.e. https://www.ni-dieu-ni-maitre.com/image_mini/xcustom-clothing-00109939891.jpg.pagespeed.ic.R_F8XM4mtg.webp
edit: now most of the images from our website are broken due to pagespeed
Lofesa
commented
5 years ago If I add this to my htaccess:
ModPagespeedFetchHttps enable,allow_self_signedI get an Internal Server Error.
Whats messages do you have when enable FechtHttps?
FechtHttps need this: # Certificate Authorities directory, not your domain SSL keys ModPagespeedSslCertDirectory directory # Web Server's HTTPS client SSL key, not your domain SSL keys ModPagespeedSslCertFile file.
and take care of:
These directive cannot be used in .htaccess files or \
I think you get the 404 error cause pagespeed can´t fecht https resources, and I think that cause the internal server error. Can you try the LoadFromFile ?
AnonDev1312
commented
5 years ago Whats messages do you have when enable FechtHttps?
Just a white page displaying "Internal Server Error"
These directive cannot be used in .htaccess files or
scopes. Then where am I supposed to put these directives for Apache? The documentation is so confusing
Anyway, do I really need to do this if FetchHttps is enabled by default?
Can you try the LoadFromFile ? I tried but I failed :(
Some of our images are generated dynamically so we can't load them from disk. example:
RewriteRule ^image_mini/([^.]+)-00([0-9]+)\.jpg$ /image_mini_jpg.php?a=$2 [L,QSA,NC]
By default PageSpeed loads sub-resources via an HTTP fetch. It would be faster to load sub-resources directly from the filesystem, however this may not be safe to do because the sub-resources may be dynamically generated or the sub-resources may not be stored on the same server.
Lofesa
commented
5 years ago Just a white page displaying "Internal Server Error"
And no messages in the apache log?
These directive cannot be used in .htaccess files or scopes. Then where am I supposed to put these directives for Apache? The documentation is so confusing
Anyway, do I really need to do this if FetchHttps is enabled by default?
Yes, you need it. You can put it in the "httpd.conf" (or "apache2.conf") file in a scope other than Directory , maybe in a vhost scope. In nginx you can put this in the http block or in the server block, but not in a location block.
Can you try the LoadFromFile ? I tried but I failed :(
Some of our images are generated dynamically so we can't load them from disk. example:
RewriteRule ^image_mini/([^.]+)-00([0-9]+)\.jpg$ /image_mini_jpg.php?a=$2 [L,QSA,NC]By default PageSpeed loads sub-resources via an HTTP fetch. It would be faster to load sub-resources directly from the filesystem, however this may not be safe to do because the sub-resources may be dynamically generated or the sub-resources may not be stored on the same server.
But images are loaded from the same server? If yes, you can try some like that:
pagespeed LoadFromFile "https://$host/" "$document_root/";
pagespeed LoadFromFileRuleMatch disallow .; (This diables load any file) pagespeed LoadFromFileRuleMatch allow .(js|css|jpg|jpeg|png|pdf|gif)(\?.)?$ps_dollar; (1) (And this enable load files with these extensions, no matter what the name is)
(1).- Because Nginx uses the $-sign to indicate script variables, when you turn on ProcessScriptVariables you need to make a small change to any script-supporting commands that are using $.
P.S: You are talking about puting directives in .httacces and this file is related to Apache server, but i see in the headers the server is nginx. Maybe you are using a nginx as a proxy cache? If not then the .httaccess does nothing in a nginx server, but if yes, then you need to configure a downstream cache
AnonDev1312
commented
5 years ago Yes, you need it. You can put it in the "httpd.conf" (or "apache2.conf") file in a scope other than Directory , maybe in a vhost scope. In nginx you can put this in the http block or in the server block, but not in a location block.
This is where it gets complicated. I can't even find where are the files to edit and I don't even understand the difference between directory scope or vhosts scope. I googled some instructions but I can't find "apache2.conf" anywhere According to this: https://www.phusionpassenger.com/library/install/apache/working_with_the_apache_config_file.html The file should be in this path but it's not!! /etc/apache2/apache2.conf
But images are loaded from the same server?
Yes
Maybe you are using a nginx as a proxy cache?
Yes, but I have no idea how it is configured... We used cpnginx for the setup and we didn't have anything to do manually, it's basically plug-and-play
but if yes, then you need to configure a downstream cache
omg I don't think I will be able to successfully do this. Does this means that modpagespeed will be broken without downstream cache? ... maybe I should just give up right now and stop wasting your time ...
This is the 4th time we try to install modpagespeed on this server in the last year and every single time we had lot of bugs, spent a lot of time trying to fix them but failed and end up giving up and uninstalling it :( This time we tried installing modpagespeed on Apache instead of just Nginx because we assumed we would be able to make the changes in htaccess files with FTP, which is way easier and user-friendly, but it looks like we will still have to edit linux config files :(
AnonDev1312
commented
5 years ago I still can't find apache2.conf but I found httpd.conf in /etc/apache2/conf/ However, I can't edit this file: http://prntscr.com/o03gpn
Lofesa
commented
5 years ago Hi @anarchoi As stated in the image you can create/edit the /etc/apache2/conf.d/pre_main_global.conf then rebuild the http2.conf with the cpanel command (or with the cpanel gui if have these option), maybe you edit this file from the cpanel gui.
If you use a nginx proxy-cache in front of the apache server you need to configure the downstream cache in pagespeed config w/o these config mostly the nginx caches the unoptimized resources and serves it.
As a test... can you disable the nginx proxy cache to see how the config work in the apache server? If you can bypss the nginx proxy you can view how the pagespeed module is working in the apache server and then, when the apache is ok, intruce the nginx proxy. Maybe all the issue (or most of it) came from this.
P.D. Don´t worry about my time, i have a lot =:-))
ioweb-gr
commented
3 years ago I also face the same issue with 4xx status code preventing rewriting. I'm using the following setup
pagespeed RewriteLevel PassThrough;
pagespeed FileCachePath /var/ngx_pagespeed_cache;
pagespeed FileCacheSizeKb 204800;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed FileCacheInodeLimit 500000;
pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;
pagespeed CreateSharedMemoryMetadataCache "/var/ngx_pagespeed_cache/" 51200;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters recompress_webp;
pagespeed EnableFilters convert_jpeg_to_webp,convert_to_webp_lossless,convert_to_webp_animated;
pagespeed Statistics on;
pagespeed StatisticsLogging on;
pagespeed LogDir /var/log/ngx_pagespeed;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed UsePerVhostStatistics on;
pagespeed MessageBufferSize 100000;
pagespeed EnableCachePurge on;
pagespeed SslCertDirectory /etc/ssl/certs;
pagespeed FetchHttps enable,allow_self_signed;And also
pagespeed on;
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }
location /ngx_pagespeed_statistics { allow 127.0.0.1; deny all; }
location /ngx_pagespeed_global_statistics { allow 127.0.0.1; deny all; }
location /ngx_pagespeed_message { allow 127.0.0.1; deny all; }
location /pagespeed_console { allow 127.0.0.1; deny all; }
location ~ ^/pagespeed_admin { allow 127.0.0.1; deny all; }
location ~ ^/pagespeed_global_admin { allow 127.0.0.1; deny all; }My nginx server is working as a reverse proxy but serves static files directly, without passing them to apache. When I enable the
pagespeed LoadFromFile ...
directive, things start working but I can't do it because my static resources are using a dynamic versioning system.
Therefore the path will never be the same as the filepath.
On my website I can see the following if I enable debug mode
mod_pagespeed on
Filters:
ah Add Head
cc Combine Css
jc Combine Javascript
gp Convert Gif to Png
jp Convert Jpeg to Progressive
jw Convert Jpeg To Webp
mc Convert Meta Tags
pj Convert Png to Jpeg
ws When converting images to WebP, prefer lossless conversions
db Debug
ec Cache Extend Css
ei Cache Extend Images
es Cache Extend Scripts
fc Fallback Rewrite Css
if Flatten CSS Imports
hw Flushes html
ci Inline Css
ii Inline Images
il Inline @import to Link
ji Inline Javascript
js Jpeg Subsampling
rj Recompress Jpeg
rp Recompress Png
rw Recompress Webp
ri Resize Images
cf Rewrite Css
jm Rewrite External Javascript
jj Rewrite Inline Javascript
cu Rewrite Style Attributes With Url
cp Strip Image Color Profiles
md Strip Image Meta Data
Options:
EnableRewriting (e) 1
FileCacheInodeLimit (afcl) 500000
RunExperiment (fur) False
StatisticsLogging (asle) True
#NumFlushes 0
#EndDocument after 19760us
#Total Parse duration 15459us
#Total Render duration 26380us
#Total Idle duration 4301usOn the html code I see full messages like
4xx status code, preventing rewriting of https://ww.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/r/wrt106300_a.jpg
Lofesa
commented
3 years ago Hi
| pagespeed LoadFromFile ...
|directive, things start working but I can't do it because my static resources are using a dynamic versioning system.
But you have a "fixed" part:
pagespeed LoadFromFile "https://www.example.com/pub/media/" "/path-to-the fixed-one/pub/media/"
Anyway, if load from file works, the fail is in the https fecht.
I can´t see a pagespeed Domain http*://*.domain.com
By default pagespeed authorizes the domain from where resources are served but only htttp no https. Whis the given directive you cover both http and https and all subdomains.
In the log files can you see any request made with UA with "Serf" in it?
In the file cache (/var/ngx_pagespeed_cache) under v3 folder what you see?
ioweb-gr
commented
3 years ago But you have a "fixed" part: pagespeed LoadFromFile "https://www.example.com/pub/media/" "/path-to-the fixed-one/pub/media/"
Not always, some paths will be rendered like this
pub/static/version1636311072/frontend/...../image.png
I can´t see a pagespeed Domain http://.domain.com
I just added it as
pagespeed Domain http*://*.subdomain.example.com;
and as
pagespeed Domain http*://subdomain.example.com;
and as
pagespeed Domain http*://*.example.com;
but I don't see any change in the errors :(
Here's what shows in pagespeed admin in the first case pagespeed Domain http*://*.subdomain.example.com; for example
Version: 14: on
Filters
gp Convert Gif to Png
jp Convert Jpeg to Progressive
jw Convert Jpeg To Webp
pj Convert Png to Jpeg
wa Convert animated images to WebP
ws When converting images to WebP, prefer lossless conversions
hw Flushes html
js Jpeg Subsampling
rj Recompress Jpeg
rp Recompress Png
rw Recompress Webp
cp Strip Image Color Profiles
md Strip Image Meta Data
Options
AdminPath (nap) /pagespeed_admin
AvoidRenamingIntrospectiveJavascript (aris) True
BeaconUrl (bu) /ngx_pagespeed_beacon
ConsolePath (ncp) /pagespeed_console
EnableCachePurge (euci) True
EnableRewriting (e) 1
FetchHttps (fhs) enable,allow_self_signed
FileCacheCleanIntervalMs (afcci) 3600000
FileCacheInodeLimit (afcl) 500000
FileCachePath (afcp) /var/ngx_pagespeed_cache
FileCacheSizeKb (afc) 204800
GlobalAdminPath (ngap) /pagespeed_global_admin
GlobalStatisticsPath (ngsp) /ngx_pagespeed_global_statistics
LogDir (ald) /var/log/ngx_pagespeed
LRUCacheByteLimit (alcb) 16384
LRUCacheKbPerProcess (alcp) 8192
MessagesPath (nmp) /ngx_pagespeed_message
RewriteLevel (l) Pass Through
SslCertDirectory (assld) /etc/ssl/certs
Statistics (ase) True
StatisticsLogging (asle) True
StatisticsPath (nsp) /ngx_pagespeed_statistics
Domain Lawyer
http*://*.subdomain.example.com/ AuthIn the log files can you see any request made with UA with "Serf" in it?
I grepped 'serf' in the nginx access logs and I can see it if you mean that record
../logs/proxy_error_log:2021/11/08 10:42:10 [error] 1559750#0: [ngx_pagespeed 1.13.35.2-0] http://subdomain.example.com/pub/media/icons/34xNxetennis_rating_banner.png (connecting to:localhost):0:serf_context_run error status=111 (Connection refused)In the file cache (/var/ngx_pagespeed_cache) under v3 folder what you see?
I can see files for my server IP and my main domain (the site is in a subdomain however I see only the domain part if that's correct) , I marked with an arrow my domain
Lofesa
commented
3 years ago Not always, some paths will be rendered like this
pub/static/version1636311072/frontend/...../image.png
But you still have a "fixed" part, until the/pub/ folder. I will say if all statics resources hosted in the local nginx disk are under 1 or 2 folders you can do LoadFromFile with 1 or 2 directives,
I can´t see a pagespeed Domain http://.domain.com
I just added it as
pagespeed Domain http*://*.subdomain.example.com;and as
pagespeed Domain http*://subdomain.example.com;and as
pagespeed Domain http*://*.example.com;
This last directive cover all others.
but I don't see any change in the errors :(
You need to clear the pagespeed cahe
Here's what shows in pagespeed admin in the first case
pagespeed Domain http*://*.subdomain.example.com;for exampleVersion: 14: on Filters gp Convert Gif to Png jp Convert Jpeg to Progressive jw Convert Jpeg To Webp pj Convert Png to Jpeg wa Convert animated images to WebP ws When converting images to WebP, prefer lossless conversions hw Flushes html js Jpeg Subsampling rj Recompress Jpeg rp Recompress Png rw Recompress Webp cp Strip Image Color Profiles md Strip Image Meta Data Options AdminPath (nap) /pagespeed_admin AvoidRenamingIntrospectiveJavascript (aris) True BeaconUrl (bu) /ngx_pagespeed_beacon ConsolePath (ncp) /pagespeed_console EnableCachePurge (euci) True EnableRewriting (e) 1 FetchHttps (fhs) enable,allow_self_signed FileCacheCleanIntervalMs (afcci) 3600000 FileCacheInodeLimit (afcl) 500000 FileCachePath (afcp) /var/ngx_pagespeed_cache FileCacheSizeKb (afc) 204800 GlobalAdminPath (ngap) /pagespeed_global_admin GlobalStatisticsPath (ngsp) /ngx_pagespeed_global_statistics LogDir (ald) /var/log/ngx_pagespeed LRUCacheByteLimit (alcb) 16384 LRUCacheKbPerProcess (alcp) 8192 MessagesPath (nmp) /ngx_pagespeed_message RewriteLevel (l) Pass Through SslCertDirectory (assld) /etc/ssl/certs Statistics (ase) True StatisticsLogging (asle) True StatisticsPath (nsp) /ngx_pagespeed_statistics Domain Lawyer http*://*.subdomain.example.com/ AuthIn the log files can you see any request made with UA with "Serf" in it?
I grepped 'serf' in the nginx access logs and I can see it if you mean that record
../logs/proxy_error_log:2021/11/08 10:42:10 [error] 1559750#0: [ngx_pagespeed 1.13.35.2-0] http://subdomain.example.com/pub/media/icons/34xNxetennis_rating_banner.png (connecting to:localhost):0:serf_context_run error status=111 (Connection refused)
This can explain the 404, pagespeed is unable to connect because is refused. Take a look, the request is made as http not https
Maybe you need to set:
pagespeed RespectXForwardedProto on;
Because I think you have some like:
client -> https -> nginx+pagespeed -> http -> localhost
In the file cache (/var/ngx_pagespeed_cache) under v3 folder what you see?
I can see files for my server IP and my main domain (the site is in a subdomain however I see only the domain part if that's correct) , I marked with an arrow my domain
Is your domain folder under some folder likehttp,3A or under https,3A?
If underhttp,3A, optimized resources are stored herem but you are requesting a optimized resource underhttps,3A and the resource is not here, so 404....
ioweb-gr
commented
3 years ago This last directive cover all others.
OK I'm sticking with just this then
You need to clear the pagespeed cahe
I cleared the cache from purge cache in pagespeed admin.
This can explain the 404, pagespeed is unable to connect because is refused. Take a look, the request is made as http not https Maybe you need to set: pagespeed RespectXForwardedProto on;
I've just enabled this option
Version: 14: on
Filters
gp Convert Gif to Png
jp Convert Jpeg to Progressive
jw Convert Jpeg To Webp
pj Convert Png to Jpeg
wa Convert animated images to WebP
ws When converting images to WebP, prefer lossless conversions
hw Flushes html
js Jpeg Subsampling
rj Recompress Jpeg
rp Recompress Png
rw Recompress Webp
cp Strip Image Color Profiles
md Strip Image Meta Data
Options
AdminPath (nap) /pagespeed_admin
AvoidRenamingIntrospectiveJavascript (aris) True
BeaconUrl (bu) /ngx_pagespeed_beacon
ConsolePath (ncp) /pagespeed_console
EnableCachePurge (euci) True
EnableRewriting (e) 1
FetchHttps (fhs) enable,allow_self_signed
FileCacheCleanIntervalMs (afcci) 3600000
FileCacheInodeLimit (afcl) 500000
FileCachePath (afcp) /var/ngx_pagespeed_cache
FileCacheSizeKb (afc) 204800
GlobalAdminPath (ngap) /pagespeed_global_admin
GlobalStatisticsPath (ngsp) /ngx_pagespeed_global_statistics
LogDir (ald) /var/log/ngx_pagespeed
LRUCacheByteLimit (alcb) 16384
LRUCacheKbPerProcess (alcp) 8192
MessagesPath (nmp) /ngx_pagespeed_message
RespectXForwardedProto (rxfp) True
RewriteLevel (l) Pass Through
SslCertDirectory (assld) /etc/ssl/certs
Statistics (ase) True
StatisticsLogging (asle) True
StatisticsPath (nsp) /ngx_pagespeed_statistics
Domain Lawyer
http*://*.example.com/ Auth
Invalidation Timestamp: Mon, 08 Nov 2021 11:34:39 GMT (1636371279251)Is your domain folder under some folder like http,3A or under https,3A? If under http,3A, optimized resources are stored herem but you are requesting a optimized resource under https,3A and the resource is not here, so 404....
Yes they are and indeed I can see some webp files there
ll /var/ngx_pagespeed_cache/v3/example.com/https,3A/,2Fsubdomain.example.com/pub/media/
catalog/
favicon/
icons/
porto/
rackets_adults_bg_eng.png,
rackets_kids_bg_eng.png,
revslider/
shoes_bg.png,
shoes_running_bg_eng.png,
shoes_tennis_bg_eng.png,
wysiwyg/
xrackets_adults_bg_eng.png.pagespeed.ic.1hFd9psVc_.webp,
xrackets_adults_bg_eng.png.pagespeed.ic.wrorOf4Pul.jpg,
xrackets_kids_bg_eng.png.pagespeed.ic.45BNnmwr_f.png,
xrackets_kids_bg_eng.png.pagespeed.ic.xabPGK8I2V.webp,
xshoes_bg.png.pagespeed.ic.KR4WNpIxi0.jpg,
xshoes_bg.png.pagespeed.ic.PqTyvlBx84.webp,
xshoes_running_bg_eng.png.pagespeed.ic.eU8V0SXHN9.jpg,
xshoes_running_bg_eng.png.pagespeed.ic.sEHw6g3I87.webp,
xshoes_tennis_bg_eng.png.pagespeed.ic.elCjgJ61XN.webp,
xshoes_tennis_bg_eng.png.pagespeed.ic.oIjD0NVeCW.png,
xxshoes_running_bg_eng.png,2CMic.sEHw6g3I87.webp.pagespeed.ic.lMgFnzIr9n.webp,But nope they're not getting served as webp. I no longer see the error for http though
Lofesa
commented
3 years ago But nope they're not getting served as webp. I no longer see the error for http though
And still have the 404?
Not serves as webp, but are rewrited by pagespeed? The url have pagespeed in it? like:
xshoes_tennis_bg_eng.png.pagespeed.ic.oIjD0NVeCW.png
To serve webp images, the browser must be webp capable, in the request made must have a header like:
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
If the request don´t have image/web in the accept header, no webp is served.
ioweb-gr
commented
3 years ago Yes the request for each image has the header for webp
And I still see this in the comments
ioweb-gr
commented
3 years ago Not serves as webp, but are rewrited by pagespeed? The url have pagespeed in it? like:
xshoes_tennis_bg_eng.png.pagespeed.ic.oIjD0NVeCW.png
No they don't :(
Lofesa
commented
3 years ago Ummm In the image I see a cache-control header:
cache-control: s-maxage=10
but pagespeed need resources public cacheable
cache-control: max-age=600 or cache-control: max-age=600,public
Then the optimized resource is stored in the cache the amount of time in this header, so in the example, the optimized resource expires in 600 sec. and after that the optimization proccess re-start.
Optimized resources have a cache-control for 1 year.
Lofesa
commented
3 years ago But the thing is the 404. optimized resources are in the https cache, so why the 404?
Test again the log and see if request whit "Serf" in the UA are here. Some like
"GET /logo_sticky.png HTTP/1.1" 404 577 "http://beautivmazure.cloudapp.net/" "Serf/1.1.0 mod_pagespeed/1.9.32.14-0" *0*
ioweb-gr
commented
3 years ago The problem is with the current setup, I don't see anything in the proxy_error_log like before and there's no Serf match in any log, not nginx access or error log, not apache access or error log either.
Version: 14: on
Filters gp Convert Gif to Png jp Convert Jpeg to Progressive jw Convert Jpeg To Webp pj Convert Png to Jpeg wa Convert animated images to WebP ws When converting images to WebP, prefer lossless conversions hw Flushes html js Jpeg Subsampling rj Recompress Jpeg rp Recompress Png rw Recompress Webp cp Strip Image Color Profiles md Strip Image Meta Data
Options AdminPath (nap) /pagespeed_admin AvoidRenamingIntrospectiveJavascript (aris) True BeaconUrl (bu) /ngx_pagespeed_beacon ConsolePath (ncp) /pagespeed_console EnableCachePurge (euci) True EnableRewriting (e) 1 FetchHttps (fhs) enable,allow_self_signed FileCacheCleanIntervalMs (afcci) 3600000 FileCacheInodeLimit (afcl) 500000 FileCachePath (afcp) /var/ngx_pagespeed_cache FileCacheSizeKb (afc) 204800 GlobalAdminPath (ngap) /pagespeed_global_admin GlobalStatisticsPath (ngsp) /ngx_pagespeed_global_statistics LogDir (ald) /var/log/ngx_pagespeed LRUCacheByteLimit (alcb) 16384 LRUCacheKbPerProcess (alcp) 8192 MessagesPath (nmp) /ngx_pagespeed_message RespectXForwardedProto (rxfp) True RewriteLevel (l) Pass Through SslCertDirectory (assld) /etc/ssl/certs Statistics (ase) True StatisticsLogging (asle) True StatisticsPath (nsp) /ngx_pagespeed_statistics
Domain Lawyer http://.example.com/ Auth
ioweb-gr
commented
3 years ago The weird thing is that on pagespeed_console, everything is 0
Lofesa
commented
3 years ago And what you can see in the pagespeed "Message History"?
https://example.com/pagespeed_global_admin/message_history is the url you can view log messages from pagespeed.
ioweb-gr
commented
3 years ago For example for some jpg files
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson-roland-garros-starter-green-junior-tennis-balls-x-4.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/i/wilson-roland-garros-starter-green-junior-tennis-balls-x-4.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/r/wrx2096.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/r/wrx2096.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/4/7/474182-tretorn-jumbo-tennis-ball-yellow-m.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/4/7/474182-tretorn-jumbo-tennis-ball-yellow-m.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/b/a/babolat-team-tennis-balls-x-4.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/b/a/babolat-team-tennis-balls-x-4.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/c/o/code-red-tennis_balls.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/c/o/code-red-tennis_balls.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/c/o/code-black-tennis_balls.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/c/o/code-black-tennis_balls.jpg fragment=example.com: remembering recent failure for 134 seconds.
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] Trying to serve rewritten resource in-place: https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/4/7/474410-75-tretorn-academy-red-36-balls-red.jpg
[Mon, 08 Nov 2021 12:36:15 GMT] [Info] [1698645] HTTPCache key=https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/4/7/474410-75-tretorn-academy-red-36-balls-red.jpg fragment=example.com: remembering recent failure for 134 seconds.Where are those messages stored in the logs?
Lofesa
commented
3 years ago These messages are stored in files, don´t remember where are stored, but are non human readable. Nothing to do whit server logs. But they says that pagespeed fail requesting the files and "remember" the fail XXX secons. If I´m not wrong you have some leke this:
client -> https -> nginx+pagespeed -> http (maybe now https) -> localhost as showed in the fecht error.
In localhost,there is an apache server? Have rewrite rules? Some like this:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^ /404 [L,R=301]Is this server runing pagespeed?
ioweb-gr
commented
3 years ago Well since PLESK autoconfigures most of them I'm not exactly sure about every detail but let me post the nginx conf and the vhost conf and maybe we can shed some light to what the issue is.
ioweb-gr
commented
3 years ago Let me note that most nginx directives for pagespeed which are common for all vhosts, are directly inserted in the nginx conf files and then I modify on a per vhost basis that's why there's a vhost_conf file for nginx and apache. So I can just tweak one vhost for what I need.
Lofesa
commented
3 years ago This config is a lot confusing for me. You have a server block for each IP/port, and is supossed that ipv4:443 must have the exact same config that ipv6:443, both are serving the same content whit the same config, no? Why you no do this:
server {
name bla.bla.com;
name www.bla.bla.com;
listen ipv4:443
listen ipv6:443
}Whit the config you have, I think all request that go to the :80 port are served by the ipv4:443 because is the 1st server blok that matches :443 and the same name
But the relevan pat for the issue is here:
location @fallback {
proxy_pass https://[2a01:4f9:3a:1041::2]:7081;
proxy_hide_header upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log off;
}
location ~^/(.*\.ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|eot|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|svg|swf|tar|tgz|ttf|txt|wav|woff|woff2|xls|xlsx|zip|webp))$ {
try_files $uri @fallback;
}If the file is not found then go to the apache. Can you see if in the apache logs are request for url with pagespeed in it? Apache don´t run pagespeed, rigth? so when a url with pagespeed in it arrives then return a 404.
I I understand you, the pagespeed directives are included in. `include "/var/www/vhosts/system/subdomain.example.com/conf/vhost_nginx.conf";
put this line at the top of the server block or allmost before the location for statics files. Why?
Because uri URI like
/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/w/r/xwrt106300_a.jpg.pagespeed.ic.SOMEHASH.webp
is captured by the location, so never reaches the locations you have in the vhos_nginx.conf, this one
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
add_header "" "";
}`Is how nginx works, parse the config file up to down, and when some matches stop the search.
ioweb-gr
commented
3 years ago Indeed I could find some apache 404s
grep -ri '\.pagespeed' ../logs/access_ssl_log* |
grep -i ' 404 '
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:26 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/Anowave_Ec,_js,_ec.js+js,_stores.js+Vendor_Customizer,_js,_bodyclass.js+Vendor_Customizer,_js,_stickyheader.js+Vendor_Customizer,_js,_popupcontent.js+Vendor_Wcag,_js,_wcag.js.pagespeed.jc.joHtLkNy1p.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:28 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/Anowave_Ec,_js,_ec.js+js,_stores.js+Vendor_Customizer,_js,_bodyclass.js+Vendor_Customizer,_js,_stickyheader.js+Vendor_Customizer,_js,_popupcontent.js+Vendor_Wcag,_js,_wcag.js.pagespeed.jc.joHtLkNy1p.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:49 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/mage,_requirejs,_mixins.js+requirejs-config.js.pagespeed.jc.I3Me962xHP.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:49 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/Anowave_Ec,_js,_ec.js+js,_stores.js+Vendor_Customizer,_js,_bodyclass.js+Vendor_Customizer,_js,_stickyheader.js+Vendor_Customizer,_js,_popupcontent.js+Vendor_Wcag,_js,_wcag.js.pagespeed.jc.joHtLkNy1p.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:50 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/mage,_requirejs,_mixins.js+requirejs-config.js.pagespeed.jc.I3Me962xHP.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
../logs/access_ssl_log.processed:37.6.136.255 - - [08/Nov/2021:16:25:50 +0100] "GET /pub/static/version1636311072/frontend/Vendor/module/el_GR/Anowave_Ec,_js,_ec.js+js,_stores.js+Vendor_Customizer,_js,_bodyclass.js+Vendor_Customizer,_js,_stickyheader.js+Vendor_Customizer,_js,_popupcontent.js+Vendor_Wcag,_js,_wcag.js.pagespeed.jc.joHtLkNy1p.js HTTP/2.0" 404 370 "https://subdomain.example.com/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"So that means apache is indeed trying to handle pagespeed requests under certain cases
Lofesa
commented
3 years ago Yes, not "under certain cases" but in all cases file extension match the nginx location. URL rewrited by pagespeed have the same file extension that the original file or in a image case maybe is changed by webp that matches too. If you have: location regex expresion for file extension {.... } location for pagespeed in url {...} pagespeed uri are captured allways by the location for file extension, don´t found it as a file and then go to apache, that know nothing about pagespeed uris, so return 404. Nginx parse the config file from up to down and the location that 1s match the regex expresion captures the uri. If you have all the pagespeed config (location block included) in the vhos_nginx.conf file, you must put it before the location for file extensions. Ths makes pagespeed uris to be captured by the
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
add_header "" "";
}and not by the location for file extensions.
ioweb-gr
commented
3 years ago I struggled a bit on how to remove the location directive which precedes the pagespeed one and now I have this
On the frontend I see only a single image converted to webp but no others
ioweb-gr
commented
3 years ago On next step I'll try moving it up as you suggested
ioweb-gr
commented
3 years ago I modified and now it looks like this
But getting the same result as before
Lofesa
commented
3 years ago You still viewing pagespeed request in the apache log? Like these in https://github.com/apache/incubator-pagespeed-mod/issues/1897#issuecomment-963947963 Have you cleared the pagespeed cache?
ioweb-gr
commented
3 years ago I've modified my configuration file template so that all custom directives I write will go directly below the command
proxy_read_timeout
and above any other location blocks.
Now the result is that I'm getting 404s for some images only e.g.
pub/media/xrackets_adults_bg_eng.png.pagespeed.ic.1hFd9psVc_.webp
My apache log doesn't mention anything with pagespeed in it
grep -ri pagespeed ../logs/access_log ../logs/access_ssl_log
yields no result so I think apache is not used in this case.
But on nginx logs I see results
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:03 +0100] "GET /pub/media/wysiwyg/xespa.png.pagespeed.ic.Srrf3IOYbP.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:03 +0100] "GET /pub/media/xrackets_adults_bg_eng.png.pagespeed.ic.1hFd9psVc_.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:03 +0100] "GET /pub/static/version1636968163/frontend/Ioweb/etennis/el_GR/images/xloader-1.gif.pagespeed.ic.DaGwXwKN2x.webp HTTP/2.0" 200 40 "https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:03 +0100] "GET /pub/media/xrackets_kids_bg_eng.png.pagespeed.ic.xabPGK8I2V.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:03 +0100] "GET /pub/media/wysiwyg/xespa.png.pagespeed.ic.Srrf3IOYbP.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:15 +0100] "GET /pub/media/wysiwyg/xespa.png.pagespeed.ic.Srrf3IOYbP.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/checkout/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:15 +0100] "GET /pub/media/xrackets_adults_bg_eng.png.pagespeed.ic.1hFd9psVc_.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/checkout/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"
../logs/proxy_access_ssl_log:222.252.xx.xxx - - [15/Nov/2021:12:11:15 +0100] "GET /pub/media/xrackets_kids_bg_eng.png.pagespeed.ic.xabPGK8I2V.webp HTTP/2.0" 404 370 "https://subdomain.example.com/storeviewgreek/checkout/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"If you try some like:
https://subdomain.example.com/pub/media/wysiwyg/espa.png
what you get?
Any instance of "Serf" in the logs?
What do you see in the mesage history (https://subdomain.example.com/pagespeed_global_admin/message_history)
What do you see when enable pagespeed debut to the URI ( https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html?PageSpeedFilters=+debug and take a look in the html code for comments)
ioweb-gr
commented
3 years ago OK let me try and sort it out
I'm going to try URL
/storeviewgreek/tennis-balls.html?PageSpeedFilters=+debug
Let's answer the questions one by one
If you try some like: https://subdomain.example.com/pub/media/wysiwyg/espa.png what you get?
In nginx access ssl log
15/Nov/2021:12:43:01 +0100] "GET /pub/media/wysiwyg/espa.png HTTP/2.0" 200 17110 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
The image loads correctly in the browser but not resized / rewrited in webp
On apache I see this in the log
[15/Nov/2021:12:44:13 +0100] "GET /favicon.ico HTTP/1.0" 500 721 "/storeviewgreek/tennis-balls.html?PageSpeedFilters=+debug" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"
Any instance of "Serf" in the logs?
No instance of the word serf in nginx or apache logs
What do you see in the mesage history (https://subdomain.example.com/pagespeed_global_admin/message_history)
What do you see when enable pagespeed debut to the URI ( https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html?PageSpeedFilters=+debug and take a look in the html code for comments)
Some examples
Lofesa
commented
3 years ago [Warning] [153873] [x601313-dunlop-atp-official-tennis-balls_1.jpg:0] Resource based on https://subdomain.example.com/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpg but cannot access the original
Seem that pagespeed can´t found the original files where are supposeed they are:
/var/www/vhosts/subdomain.example.com/httpdocs/pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpg
Is this file here?
In other hand you must exclude some files from pagespeed, like woff or svg files because pagespeed do nothing whit these files and you get messages about it in the message history, add this
pagespeed Disallow *.woff
pagespeed Disallow *.svgto your config file
P.D. can you turn on any access_log off? Maybe these make no serf lines in the log files
P.D. 1. Seems like css files have not problem, can you confirm that css files got rewrited and have 200 http code?
P.D. 2. I can´t see where filters are enabled, some like
pagespeed EnableFilters whatever_filter_enabled;
By default pagespeed is configured with
pagespeed RewriteLevel CoreFilters
and this only enable these filters:
add_head
combine_css
combine_javascript
convert_meta_tags
extend_cache
fallback_rewrite_css_urls
flatten_css_imports
inline_css
inline_import_to_link
inline_javascript
rewrite_css
rewrite_images
rewrite_javascript
rewrite_style_attributes_with_url
`
Let me answer quickly the last part about CoreFilters which is easier , I'll answer the second part too when I investigate It' smy understanding that when I use PassThrough only the ones I enable are on right?
So in my general nginx configuration regarding pagespeed I have these options
pagespeed RewriteLevel PassThrough;
pagespeed FileCachePath /var/ngx_pagespeed_cache;
pagespeed FileCacheSizeKb 204800;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed FileCacheInodeLimit 500000;
pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;
pagespeed CreateSharedMemoryMetadataCache "/var/ngx_pagespeed_cache/" 51200;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters recompress_webp;
pagespeed EnableFilters convert_jpeg_to_webp,convert_to_webp_lossless,convert_to_webp_animated;
pagespeed Disallow *.woff;
pagespeed Disallow *.svg;
pagespeed Statistics on;
pagespeed StatisticsLogging on;
pagespeed LogDir /var/log/ngx_pagespeed;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed UsePerVhostStatistics on;
pagespeed MessageBufferSize 100000;
pagespeed EnableCachePurge on;
pagespeed SslCertDirectory /etc/ssl/certs;
pagespeed FetchHttps enable,allow_self_signed;
pagespeed RespectXForwardedProto on;So only the ones I enable should be active right?
ioweb-gr
commented
3 years ago The file exists
ll pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpg -h
-rw-rw-r-- 1 sys_ psacln 11K Oct 8 18:02 /pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpgSo only the ones I enable should be active right?
Yes. I´m asking because I can´t see these config in the vhos_nginx.conf file here https://github.com/apache/incubator-pagespeed-mod/issues/1897#issuecomment-968746800
Lofesa
commented
3 years ago The file exists
ll pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpg -h -rw-rw-r-- 1 sys_etennism2 psacln 11K Oct 8 18:02 /pub/media/catalog/product/cache/9cb80aaa700fbabda1d30deb1d8f7ff5/6/0/601313-dunlop-atp-official-tennis-balls_1.jpg
But this path is under
/var/www/vhosts/subdomain.example.com/httpdocs/
And what are the cache-control header?
Pagespeed need resources to be public cacheable and whit a max-age greather than 0.
This header makes the TTL for the optimized resource in the cache, so if you habe some like
cache-control: max-age=600
the optimized resource "lives" in the cache 600 secons, pass this time, resource must be re-optimized
When the resource is optimized, pagespeed changes it to 1 year for the browser.
ioweb-gr
commented
3 years ago Yes. I´m asking because I can´t see these config in the vhos_nginx.conf file here #1897 (comment)
I've put them in the http context which is affecting all vhosts, that's why you don't see it here. However the directives are indeed active from what I see in the pagespeed admin pages
Basically I wanted to globalize the settings, so various sites can use it if needed with a simple way like just add the location directives and enable the filters you want per website. Because different websites need different settings and this one only needs webp images conversion. Ultimately the vhosts configuration will enable pagespeed on a per vhost basis and activate the necessary filters. I've used this type of config with apache in the past and it worked for me so I continued with the same logic.
Regarding your next question
But this path is under /var/www/vhosts/subdomain.example.com/httpdocs/
Well all my vhosts are under /var/www/vhosts/<domain or subdomain>/httpdocs
But don't take the exact path literally on the log, I have to trim out some parts for privacy
Filesystem
Log
OK based on what you said, I decided to disallow css files too explicitly to reduce the messages and restarted nginx
Now a bunch of images got generated as webp and served , not all but a lot of them
After a couple of minutes, no webp images were generated. This is very weird :(
Here are the messages now
ioweb-gr
commented
3 years ago I forgot to respond that for an example image the headers are
HTTP/2 200 OK
server: nginx
date: Mon, 15 Nov 2021 13:27:59 GMT
content-type: image/jpeg
content-length: 8756
last-modified: Fri, 08 Oct 2021 16:02:14 GMT
etag: "61606b86-2234"
expires: Mon, 29 Nov 2021 13:27:59 GMT
cache-control: max-age=1209600
strict-transport-security: max-age=15768000; includeSubDomains
x-powered-by: PleskLin
accept-ranges: bytes
X-Firefox-Spdy: h2Well, this etag header tell me that this request is not managed by pagespeed. Optimized resources have a etag like etag: W/"0" or etag: "PSA/XXXXX" if optimized by the IPRO, an on the fly optimizer that don´t change the url but do some optimizations.
I have figured what your site is ( rewied your post) han have discovered some things, relevant to the issue:
https://subdomain.example.com/storeviewgreek/accessories/wilson-greek-flag-dampener-x-1-wr8413601.html?PageSpeedFilters=+debug
have a cache-control: max-age=0, no-cache, no-store this cause problems on the pagespeed cache, this cache don´t store the html code, but need to be public cacheable with max-age GT 0.
have a x-magento-cache-control and x-magento-cache-debug headers, this tell me that varnish is in place, because these headers are typically set by varnish config.
So you need to take a look at dowstream cache in pagespeed docs here
And now you have these messages in the debug comments <!--4xx status code, preventing rewriting of that can be producced by a lot of things, mostly https fechtes fails, but not only.
At a 1st step can you disable the varnish config? I think you need to get the web working with pagespeed before introducing varnish in the figure. Maybe there is no varnish but other intermediate cache.
ioweb-gr
commented
3 years ago I myself read the Varnish article and I wanted to note that I'm not using varnish. Just nginx as reverse proxy to apache.
Regarding the headers for the url I'm viewing request headers are
HTTP/2 200 OK
server: nginx
content-type: text/html; charset=UTF-8
x-powered-by: PHP/7.4.25
x-magento-cache-control: max-age=86400, public, s-maxage=86400
x-magento-tags: cat_c,cat_c_3,cat_c_49,cat_c_7,cat_c_12,cat_c_54,cat_c_61,cat_c_372,cat_c_853,cat_c_1014,store,cms_b,cms_b_footer-all,cat_c_p_54,cat_p_129993,cat_p,cat_p_129500,cat_p_119921,cat_p_121195,cat_p_118932,cat_p_118933,cat_p_118994,cat_p_118930,cat_p_118931,cat_p_118993,cat_p_119207,cat_p_113823,cat_p_111868,cat_p_98893,cat_p_96700,cat_p_96036,cat_p_96045,cat_p_96697,cat_p_96702,cat_p_96635,cat_p_96634,cat_p_96159,cat_p_96160,cat_p_96203,cat_p_96204,cat_p_96633,cat_p_99173,cat_p_94689,cat_p_107512,cat_p_93850,cat_p_93851,cat_p_91606,cat_p_108936,cat_p_90454,cat_p_82292,cat_p_76719,cat_p_74920,cat_p_71801,cat_p_128314,cat_p_92152,cat_p_71803,cat_p_128313,cat_p_101999,cat_p_66737,cat_p_66740,cat_p_66741,cat_p_62604,cat_p_61869,cat_p_98220,cat_p_98215,cat_p_53191,cat_p_52704,cat_p_52191,cat_p_54570,cat_p_52692,cat_p_49621,cat_p_49716,cat_p_49021,cat_p_49024,mobile,amasty_mega_menu,cms_b_etennis_greek_header_custom_block,cms_b_porto_category_side_custom_block,cms_b_porto_footer_bottom_custom_block,FPC
pragma: no-cache
x-magento-cache-debug: HIT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
set-cookie: PHPSESSID=lub80ttmofa7l5cn2itrh7kecb; expires=Mon, 15-Nov-2021 15:30:07 GMT; Max-Age=3600; path=/; domain=subdomain.example.com; secure; HttpOnly; SameSite=None
set-cookie: X-Magento-Vary=c58cc7336841735bf5ef13185766282824a9d073; expires=Mon, 15-Nov-2021 15:30:07 GMT; Max-Age=3600; path=/; secure; HttpOnly; SameSite=None
x-ua-compatible: IE=edge
strict-transport-security: max-age=15768000; includeSubDomains
x-powered-by: PleskLin
date: Mon, 15 Nov 2021 14:30:07 GMT
x-page-speed: 1.13.35.2-0
cache-control: max-age=0, no-cache, no-store
X-Firefox-Spdy: h2I'm pretty sure cache-control header is sent by magento 2 itself without varnish.
But we don't really want to cache the full pages themselves in the browser, we just need to serve optimized images and cache the static assets. That's why we add expires headers for images for example.
Could you explain to me how the cache-control in the main page is interfering with pagespeed?
x-magento-cache-control and x-magento-cache-debug headers are there because this is a staging site in developer mode. They go away in production.
ioweb-gr
commented
3 years ago Without changing any config this time I saw these images converted for example
Lofesa
commented
3 years ago Well.... think pagespeed is a proxy too, don´t store the html but some metadata about it.
When pagespeed is working, html pages are served with cache-control: max-age=0, no-cache unless you explicity the
pagespeed ModifyCachingHeaders off;
this directive tell to pagespeed to no rewrite cache-control headers.
And yes, some images got rewrited by pagespeed and some not.
These images not rewrited have the <!--4xx status code, preventing rewriting of in the debug comments and you can see the diff in headers, you can see etag, cache-control, x-pagespeed.... are different, seems like these images are untouched by pagespeed, like if they are not in the location and then go to the apache.
I found that storeviewgreek.png and espa.png images have different headers, the 1st have a last-modified that the 2nd don´t have, so seems are served from different sites or locations, despite they are from pub/media/wysiwyg/ folder.
As I can see, al images (and other files) are served under /pub folder, as a try, can you set this:
pagespeed LoadFromFile "https://subdomain.example.com/pub/" "/var/www/vhosts/subdomain.example.com/httpdocs/pub/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.jpg$;
pagespeed LoadFromFileRuleMatch allow \.jpeg$;This makes that files under /pub/ web folder are loaded from disk /var/www/vhosts/subdomain.example.com/httpdocs/pub/ but only files with the png, jpg or jpeg are loaded, any other file extension are loaded by the regular fecht. If the files with these extensions is no here you get a message in message history that the original file can´t be found.
P.D. you need to disable all font types you serve from your domain, like woff2 or ttf, and ico files too.
ioweb-gr
commented
3 years ago I didn't mention that in my OP but when I use LoadFromFile everything worked a lot better.
But because in production we deploy with a versioning system, this pretty much fails because it will never find the files in the filesystem location it will be looking at. So I'm trying to configure it without the LoadFromFile directive.
For testing purposes I enabled it again, and it's working a lot better but it doesn't help me to enable this option
ioweb-gr
commented
3 years ago I reverted to normal fetch and based on your suggestion I enabled only pngs and jpgs to reduce the message clutter
pagespeed Disallow .*; pagespeed Allow .png$; pagespeed Allow .jpg$; pagespeed Allow .jpeg$;
Mod PageSpeed debug is showing a lot of errors like these:
I tried this but it didn't work:
Any help would be appreciated