Closed icebluey closed 2 years ago
Facing the same issue. This project is abandonded though isnt' it? No commits for 2 years now
I have the same issue
Changes with nginx 1.23.0 21 Jun 2022
*) Change in internal API: now header lines are represented as linked
lists.
I suspect this is the issue here.
I have the same issue
I found "r->headers_out.cache_control.elts" in diff of nginx-1.22.0/src/http/modules/ngx_http_headers_filter_module.c and nginx-1.23.0/src/http/modules/ngx_http_headers_filter_module.c.
@oschaaf Can take alook here?
I hope this issue push an effort of developing a stable version again as the module for nginx has been in RC1 for years.
The issue with the "double compression and mess with brotli and gzip" has been around since then i had to switch off pagespeed more than a year ago due those content enconding issues that were not allowing the browser to properly render CSS files even enabling this directive ModPagespeedHttpCacheCompressionLevel 0
@dvershinin, care to do a pull request for headers.patch? ;-)
@eilandert & @Lofesa -- https://github.com/apache/incubator-pagespeed-ngx/pull/1750 has an (untested) draft that aims to resolve this. It's still a draft, because we probably need some ifdef's to make the change conditional (I guess older nginx versions swill still need the old way of manipulating cache control headers). Anyone willing to take that change for a spin? I'd be happy to hear if it works well.
Thanks @oschaaf, I've been able to compile nginx again. Going to test functionality now
Thanks @oschaaf , it compiles.
However, my build fails thanks to openresty unpatched modules (with the same problem), so I have no packages at all to test. And I need those to run on my staging environments :-(
It's compiled but i can't make js and css minification, probably images also.
I'm getting this info message for almost all resources [Mon, 27 Jun 2022 21:52:58 GMT] [Info] [19134] No permission to rewrite 'https://shopdiggers.com/wp-includes/blocks/navigation/view.min.js'
Because i have an AWS ALB in front of the instance i do all the security layer at the ALB with AWS SSL certificates, so by then traffic goes under port 80 inside the vpc
I hope someone could through some light about what is going on
This are my global settings
pagespeed CreateSharedMemoryMetadataCache "/mnt/ramdisk/pagespeedcache/" 2048000;
pagespeed DefaultSharedMemoryCacheKB 2048000;
pagespeed FileCacheSizeKb 2048000;
pagespeed FileCacheCleanIntervalMs 36000000;
pagespeed FileCacheInodeLimit 500000;
pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;
pagespeed RedisServer "XXXXXXXXXXXXXXXXXXXXX";
pagespeed RedisDatabaseIndex 5;
pagespeed RedisReconnectionDelayMs 1;
pagespeed RedisTimeoutUs 50;
pagespeed RedisTTLSec 100000;
pagespeed ImplicitCacheTtlMs 31536000000;
pagespeed LoadFromFileCacheTtlMs 31536000000;
pagespeed ImageMaxRewritesAtOnce -1;
pagespeed HttpCacheCompressionLevel 9;
pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
pagespeed IproMaxConcurrentRecordings 0;
pagespeed EnableCachePurge on;
pagespeed PurgeMethod PURGE;
pagespeed Statistics on;
pagespeed StatisticsLogging on;
pagespeed UsePerVhostStatistics on;
pagespeed LogDir /var/log/pagespeed;
pagespeed MessageBufferSize 200000;
pagespeed XHeaderValue "Optimized by Golden Gem";
pagespeed on;
pagespeed FileCachePath "/mnt/ramdisk/pagespeedcache/";
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters extend_cache;
pagespeed EnableFilters strip_image_meta_data;
pagespeed EnableFilters jpeg_subsampling;
pagespeed EnableFilters hint_preload_subresources;
pagespeed UseNativeFetcher on;
pagespeed FetcherTimeoutMs 2000;
pagespeed RewriteDeadlinePerFlushMs 5;
resolver 8.8.8.8;
This are my settings at the VHOST
server {
#listen [::]:443 ssl http2;
listen 80;
server_name [shopdiggers.com](http://shopdiggers.com/);
root /var/www/shopdiggers;
index index.php index.html index.htm;
expires $expires;
location / {
try_files $uri $uri/ /index.php?$args;
}
# Needs to exist and be writable by nginx. Use tmpfs for best performance.
pagespeed RedisReconnectionDelayMs 500;
pagespeed ImplicitCacheTtlMs 31536000000;
pagespeed LoadFromFileCacheTtlMs 31536000000;
# Ensure requests for pagespeed optimized resources go to the pagespeed handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }
pagespeed on; pagespeed MaxCombinedJsBytes 300000; pagespeed MaxCombinedCssBytes -1; pagespeed MaxSegmentLength 1500;
pagespeed CombineAcrossPaths on; pagespeed FetchHttps enable,allow_self_signed,allow_unknown_certificate_authority,allow_certificate_not_yet_valid; pagespeed MapOriginDomain http://shopdiggers.com/ https://shopdiggers.com/; pagespeed Domain shopdiggers.com; pagespeed LoadFromFile "https://shopdiggers.com/" "/var/www/shopdiggers/";
pagespeed EnableFilters strip_image_color_profile; pagespeed EnableFilters move_css_to_head,move_css_above_scripts,flatten_css_imports; pagespeed EnableFilters rewrite_javascript,rewrite_css,rewrite_images; pagespeed EnableFilters combine_javascript,combine_css; pagespeed EnableFilters collapse_whitespace,elide_attributes; pagespeed EnableFilters remove_comments; pagespeed EnableFilters inline_google_font_css; pagespeed EnableFilters sprite_images; pagespeed EnableFilters extend_cache; pagespeed EnableFilters insert_dns_prefetch; pagespeed EnableFilters trim_urls;
Is it possible that when the HTML is rewritten, ngx_pagespeeed sees the URL of that page as something other than https://shopdiggers.com? Maybe www.shopdiggers.com?
Other than that, your config looks right to me. It might also help to turn on debug:
pagespeed EnableFilters debug;
and that will annotate your HTML with more details explaining why rewriting is not happening as expected.
Links are legit with naked domain, i moved the settings to a different domain with a site with more elements to optimize. Pagespeed settings are adjusted to this particular domain. On the output below the comments enabling debug flag
href='https://goldengem.net/wp-content/plugins/wp-media-folder//assets/css/vc_style.css' type='text/css' media='all'/>
Found it!
I knew it was related to the way the SSL works
Adding this line unleashes pagespeed features pagespeed RespectXForwardedProto on;
https://github.com/apache/incubator-pagespeed-ngx/issues/1251 Thanks! @oschaaf
Nice! I should remember that for the next time this comes up :)
@jmarantz Is a know trick when you have
user -> 443 -> Proxy -> 80 -> Backend
w/o pagespeed RespectXForwardedProto on; pagespeed respond with the protocol that the proxy request..
Other software have similar issues with this, like Fail2ban. Fail2ban is a software to ban ip for bad actors and ir works reading log files, so must run in the backend server, but w/o apropiate headers, the log file only record the proxy ip. In this case headers like X-Real-IP and/or X-Forwarded-For must be inplace and then the real user ip gets recorden in the backend server.
https://cdn.surfcampmaldives.com/wp-content/themes/uncode/library/js/init.js?ver=2122898389' id='uncode-init-js'>
Any idea why it's not optimizing resources from the CDN?
You can try to authorize that domain with:
pagespeed Domain https://cdn.surfcampmaldives.com;
This will result in the browser attempting to fetch something like https://cdn.surfcampmaldives.com/wp-content/themes/uncode/library/js/init.js,qver=2122898389.pagespeed.HASH.jm.js
That should work if your CDN is set up to pull content from your nginx server where the URL would be decoded successfully.
It doesnt work, i already had that directive set up i'm trying with this: pagespeed MapOriginDomain localhost https://cdn.surfcampmaldives.com;
https://cdn.surfcampmaldives.com/wp-content/themes/uncode/library/js/init.js?ver=114949620' id='uncode-init-js'>
These 'three guys' do the job integrating pagespeed with cloudfront. https://surfcampmaldives.com/
pagespeed MapOriginDomain localhost https://cdn.surfcampmaldives.com;
pagespeed Domain surfcampmaldives.com;
pagespeed LoadFromFile "https://cdn.surfcampmaldives.com" "/var/www/wordpress/";
@danidorado Normally I use some like
pagespeed Domain http*://*.surfcampmaldives.com;
This covers both versions, http and https and all subdomains By default pagespeed Domain only covers the http version
Take care of pagespeed LoadFromFile. When you use it, there is no http request so resources have no http headers. If pagespeed don't see any cache-control header, then apply a default of 300 (if I remember) so the optimized resource only lives these 300s in the pagespeed cache.
You need to addç
pagespeed LoadFromFileCacheTtlMs implicit_cache_ttl_in_milliseconds;
Or any other custom header (those added to request with the nginx add_header), with
pagespeed AddResourceHeader
In the config you posted I can't see
pagespeed FetchHttps enable; pagespeed SslCertDirectory directory; pagespeed SslCertFile file;
https://www.modpagespeed.com/doc/https_support#configuring_ssl_certificates
So maybe these are causing troubles.
@Lofesa
Thanks for this tip pagespeed Domain http://.surfcampmaldives.com; I think i might run into issues in the past because not having this http followed by the wildcard
I had this directive set up inside the http block at nginx.conf file pagespeed LoadFromFileCacheTtlMs 31536000000;
if i'm not wrong having that TTL set then I could skip doing this part you mentioned, right? _Or any other custom header (those added to request with the nginx addheader), with pagespeed AddResourceHeader
I read on the documentation pagespeed FetchHttps enable; is enabled by default but i've also added to the set up at the http nginx block
In relation to this two directives: pagespeed SslCertDirectory directory; pagespeed SslCertFile file;
I'm exactly doing what you mentioned here
AWS ACM -> ALB -> EC2 -> NGINX
user -> 443 -> Proxy -> 80 -> Backend
So i'm not installing certs directly on the instance, instead they are attached to the ALB through AWS console so i guess RespectXForwardedProto resolves that, i have this directive present at the http block as well pagespeed RespectXForwardedProto on;
And btw, is there anyone willing to spend 30-60 minutes on a call to review the integration and give some feedback? -open to another suitable way to do this without taking too much time of your lives- I'm happy to pay for that time if neccessary. I have some doubts if i have properly set up the caching mechanisms and also related to the data shown at the pagespeed console panel.
Thanks for letting me know if i'm doing it right.
@danidorado
I had this directive set up inside the http block at nginx.conf file pagespeed LoadFromFileCacheTtlMs 31536000000; That's ok. I mentioned this because I don't see it in the config you posted. Only one apreciation, maybe 1 year is too much time. That's supposed to be the time pagespeed stores the optimized resource in their cache . Forgot the LoadFromFile stuff.... for a momment. If you serve a resource with 1 week as cache-control header and is optimized by pagespeed, 2 things go on: 1.- The cache-control for the optimized resource is changed to 1 year (unless you configured pagespeed to preserver the original header) 3.- The original cache-control, 1 week in the example, is taken by pagespeed as the time the resource lives in their cache.
if i'm not wrong having that TTL set then I could skip doing this part you mentioned, right? Or any other custom header (those added to request with the nginx add_header), with pagespeed AddResourceHeader
Ummm... Not. That only affect cache-control. If you whish to add a custom header to the resource ( say strict-transport as example) this must be added with pagespeed AddResourceHeader
.
As far as the resource is loaded from disk and don't have any http headers, when the optimized resource is serverd it only have the default headers pagespeed put in the optimized resource, and strict-transport is not in these defaults, and don't have it even you set an add_header nginx directive.
pagespeed SslCertDirectory directory; pagespeed SslCertFile file;
You are rigth, don't need these directives as far as the request for an optimized resource not file based goes by http, not https.
This is what i typically have inside a server block for a particular domain
location ~ .php$ { include fastcgi_params; fastcgi_pass unix:/run/php-fpm/www.sock; fastcgi_intercept_errors off; fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name;
#NGINX FASTCGI-CACHE-HEADERS
fastcgi_cache_bypass $skip_cache;
fastcgi_no_cache $skip_cache;
fastcgi_cache phpcache;
fastcgi_cache_valid 200 301 302 7d;
add_header X-FastCGI-Cache $upstream_cache_status;
#SECURITY HEADERS
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
add_header X-XSS-Protection "1; mode=block" always;
fastcgi_hide_header X-Powered-By;
proxy_pass_header Server;
add_header Referrer-Policy "no-referrer-when-downgrade";
#add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval';" always;
}
location ~* \.(?:ico|css|js|gif|jpe?g|png|jpg|woff|woff2|ttf|mp4|mov|ogg)$ {
expires 366d;
#expires 1s;
add_header Pragma public;
add_header Cache-Control "public, max-age=31536000";
access_log off;
log_not_found off;
}
Well ..... php stuff is not loaded by LoadFromFile
And Pragma header I think is in the default headers pagespeed set.
But in the statics files, think you need to add, for example, a x-content-type-options: nosniff, and you add in the location bloc
add_header x-content-type-options nosniff;
This header is not set in the optimized resources. If you wish to add it to a optimized resource, you need to use
pagespeed AddResourceHeader "x-content-type-options" "nosniff";
Thanks for the heads up, i've added that directive to the list.
Would you mind if i send you an email with the htpasswd credentials to my pagespeed console and give it a look to know if all caching mechanism are set up correctly?
what about this?
needs to be added at pagespeed as well?
location ~* \.(eot|otf|ttf|woff|woff2)$ {
add_header Access-Control-Allow-Origin *;
}
what about this?
needs to be added at pagespeed as well?
location ~* \.(eot|otf|ttf|woff|woff2)$ { add_header Access-Control-Allow-Origin *; }
Not. As far as pagespeed do nothing with these files (same with .svg and .ico files)
Thanks for the heads up, i've added that directive to the list.
Would you mind if i send you an email with the htpasswd credentials to my pagespeed console and give it a look to know if all caching mechanism are set up correctly?
I have an email in the profile.....
Is there anyway to enable serialization and compression when using pagespeed and redis? In two days has filled my redis instance I have set redis for wordpress with igbinary and lz4 support so a redis DDBB of 1GB is typically now around 200MB
Any other better way to do it not involving redis?
Thanks @oschaaf, I've been able to compile nginx again. Going to test functionality now
How exactly were you able to compile this now? I'm a noob an very confused with this entire conversation. I raised a similar (if not the same) issue here #1756, and was told I might find the answer here. But I'm not finding anything. Would you be able to guide me in the right direction as to how to compile this using the patched aarch64 PSOL libraries?
It's compiled but i can't make js and css minification, probably images also.
I'm getting this info message for almost all resources [Mon, 27 Jun 2022 21:52:58 GMT] [Info] [19134] No permission to rewrite ' https://shopdiggers.com/wp-includes/blocks/navigation/view.min.js'
Because i have an AWS ALB in front of the instance i do all the security layer at the ALB with AWS SSL certificates, so by then traffic goes under port 80 inside the vpc
I hope someone could through some light about what is going on
This are my global settings
#pagespeed CreateSharedMemoryMetadataCache
"/mnt/ramdisk/pagespeedcache/" 1000000;
#pagespeed DefaultSharedMemoryCacheKB 150000;
pagespeed FileCacheSizeKb 2048000;
pagespeed FileCacheCleanIntervalMs 36000000;
pagespeed FileCacheInodeLimit 500000;
pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;
pagespeed RedisServer "XXXXXXXXXXXXXXXXXXXXX";
pagespeed RedisDatabaseIndex 5;
pagespeed RedisReconnectionDelayMs 1;
pagespeed RedisTimeoutUs 50;
pagespeed RedisTTLSec 100000;
pagespeed ImplicitCacheTtlMs 31536000000;
pagespeed LoadFromFileCacheTtlMs 31536000000;
pagespeed ImageMaxRewritesAtOnce -1;
pagespeed HttpCacheCompressionLevel 9;
pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
pagespeed IproMaxConcurrentRecordings 0;
pagespeed EnableCachePurge on;
pagespeed PurgeMethod PURGE;
pagespeed Statistics on;
pagespeed StatisticsLogging on;
pagespeed UsePerVhostStatistics on;
pagespeed LogDir /var/log/pagespeed;
pagespeed MessageBufferSize 200000;
pagespeed XHeaderValue "Optimized by Golden Gem";
pagespeed on;
pagespeed FileCachePath "/mnt/ramdisk/pagespeedcache/";
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters extend_cache;
pagespeed EnableFilters strip_image_meta_data;
pagespeed EnableFilters jpeg_subsampling;
pagespeed EnableFilters hint_preload_subresources;
pagespeed UseNativeFetcher on;
pagespeed FetcherTimeoutMs 2000;
pagespeed RewriteDeadlinePerFlushMs 5;
resolver 8.8.8.8;
This are my settings at the VHOST
server {
#listen 443 ssl http2;
#listen [::]:443 ssl http2;
listen 80;
server_name shopdiggers.com;
root /var/www/shopdiggers;
index index.php index.html index.htm;
expires $expires;
location / {
try_files $uri $uri/ /index.php?$args;
}
pagespeed RedisReconnectionDelayMs 500;
pagespeed ImplicitCacheTtlMs 31536000000;
pagespeed LoadFromFileCacheTtlMs 31536000000;
# Ensure requests for pagespeed optimized resources go to the pagespeed
handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }
pagespeed on;
pagespeed MaxCombinedJsBytes 300000;
pagespeed MaxCombinedCssBytes -1;
pagespeed MaxSegmentLength 1500;
pagespeed CombineAcrossPaths on;
pagespeed FetchHttps enable,allow_self_signed,allow_unknown_certificate_authority,allow_certificate_not_yet_valid;
pagespeed MapOriginDomain http://shopdiggers.com https://shopdiggers.com;
pagespeed Domain shopdiggers.com;
pagespeed LoadFromFile "https://shopdiggers.com" "/var/www/shopdiggers/";
pagespeed EnableFilters strip_image_color_profile;
pagespeed EnableFilters move_css_to_head,move_css_above_scripts,flatten_css_imports;
pagespeed EnableFilters rewrite_javascript,rewrite_css,rewrite_images;
pagespeed EnableFilters combine_javascript,combine_css;
pagespeed EnableFilters collapse_whitespace,elide_attributes;
pagespeed EnableFilters remove_comments;
pagespeed EnableFilters inline_google_font_css;
pagespeed EnableFilters sprite_images;
pagespeed EnableFilters extend_cache;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters trim_urls;
On Tue, Jun 28, 2022 at 4:23 AM Thijs Eilander @.***> wrote:
ok, no sleep! fixed some things.
Packages here: http://edge.deb.myguard.nl:8888/pool/main/n/nginx/ http://edge.deb.myguard.nl:8888/openssl3/pool/main/n/nginx/ (building in progress right now. please note: edge.deb.myguard.nl is experimental)
Docker here https://hub.docker.com/r/eilandert/nginx-modsecurity3-pagespeed (currently only the debian ones are with 1.23.0, when all packages are built ubuntu will follow tomorrow)
Some documentation here: https://deb.myguard.nl/nginx-modules/
Happy testing
— Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1749#issuecomment-1168029872, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTZ2OMROI6AXFG5REB7PHLVRIZXNANCNFSM5ZFWIKDA . You are receiving this because you commented.Message ID: @.***>
Nginx version: nginx 861f076eab72