Please answer to this to shed some light

[journeytosilius]

It would be cool if you can check this thread, as I think many people will benefit from it, as I am struggling configuring to make pagespeed to work with Varnish 4 :

http://serverfault.com/questions/637108/how-to-make-varnish-4-0-work-with-nginx-and-pagespeed

Thank you

[oschaaf]

I commented at the serverfault.com question - let me know if that addresses your issue.

[journeytosilius]

Hi Otto, thank you so much for your answer.

I have added the missing config in pagespeed ( silly me, I missed that ) and removed return (deliver) from the 6th block in Varnish 4.0 config )

Now Varnish is serving the optimized pages, but there are no hits to its cache, it just bypasses Varnish cache and serves the optimized pages. Do you know how to approach this to both get varnish to hit the cache and also serve the optimized pages ? Those are my headers :

HTTP/1.1 200 OK Server: nginx/1.7.6 Content-Type: text/html; charset=UTF-8 Vary: Accept-Encoding X-Powered-By: PHP/7.0.0-dev Set-Cookie: _icl_current_language=en; expires=Fri, 17-Oct-2014 07:30:35 GMT; Max-Age=86400; path=/ X-Pingback: http://domain.com/xmlrpc.php Link: http://domain.com/; rel=shortlink Access-Control-Allow-Headers: X-Requested-With Access-Control-Allow-Methods: GET, HEAD, OPTIONS Access-Control-Allow-Origin: * Date: Thu, 16 Oct 2014 07:30:36 GMT X-Page-Speed: 1.9.32.1-4238 Cache-Control: no-cache, max-age=0 X-Varnish: 131202 Age: 0 Via: 1.1 varnish-v4 X-Cache: MISS Connection: keep-alive

Do you think it has something to do with the language cookie ?

Thank you and congrats for such a nice project !

Regards

Xavier

2014-10-15 21:05 GMT+02:00 Otto van der Schaaf notifications@github.com:

I commented at the serverfault.com question - let me know if that addresses your issue.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-59258367 .

[oschaaf]

Re: "Do you think it has something to do with the language cookie ?" Yes. I think you'd need to (carefully) add some vcl to override that behaviour.

[journeytosilius]

Thank you. So, pagespeed + varnish4 confirmed working. I will check further for the language cookie issue.

Keep up the amazing work, thank you again !

Xavier

2014-10-16 11:32 GMT+02:00 Otto van der Schaaf notifications@github.com:

Re: "Do you think it has something to do with the language cookie ?" Yes. I think you'd need to (carefully) add some vcl to override that behaviour.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-59336555 .

[oschaaf]

You could confirm that the downstream cache setup is working by serving a cacheable plain html file without cookies from nginx that gets optimized by ngx_pagespeed. If you refresh a few times on that page, you should end up seeing cache hits.

[journeytosilius]

Got you. Will try that, because actually I just removed the lang cookie and it still does not work !

Will keep posting results.

Thanks

2014-10-16 11:59 GMT+02:00 Otto van der Schaaf notifications@github.com:

You could confirm that the downstream cache setup is working by serving a cacheable plain html file without cookies from nginx that gets optimized by ngx_pagespeed. If you refresh a few times on that page, you should end up seeing cache hits.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-59339412 .

[journeytosilius]

Tested with a plain html file processed by pagespeed, varnish keeps on not hitting the cache so I guess the problem resides somewhere else on the config. It is not a cookie related problem. What should be the next check on the config ?

The headers for thr html plain file :

HTTP/1.1 200 OK Server: nginx/1.7.6 Content-Type: text/html Vary: Accept-Encoding Access-Control-Allow-Headers: X-Requested-With Access-Control-Allow-Methods: GET, HEAD, OPTIONS Access-Control-Allow-Origin: * Date: Thu, 16 Oct 2014 10:22:00 GMT X-Page-Speed: 1.9.32.1-4238 Cache-Control: no-cache, max-age=0 X-Varnish: 230883 Age: 0 Via: 1.1 varnish-v4 X-Cache: MISS Connection: keep-alive

2014-10-16 12:02 GMT+02:00 Xavier Hoffman xavier@eo10.net:

Got you. Will try that, because actually I just removed the lang cookie and it still does not work !

Will keep posting results.

Thanks

2014-10-16 11:59 GMT+02:00 Otto van der Schaaf notifications@github.com:

You could confirm that the downstream cache setup is working by serving a cacheable plain html file without cookies from nginx that gets optimized by ngx_pagespeed. If you refresh a few times on that page, you should end up seeing cache hits.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-59339412 .

[oschaaf]

@chromafunk what do the response headers look like when you directly request the page from nginx instead of varnish?

[journeytosilius]

curl -I 127.0.0.1:8080/nocook.html

HTTP/1.1 200 OK Server: nginx/1.7.6 Content-Type: text/html Connection: keep-alive Vary: Accept-Encoding Access-Control-Allow-Headers: X-Requested-With Access-Control-Allow-Methods: GET, HEAD, OPTIONS Access-Control-Allow-Origin: * Date: Thu, 16 Oct 2014 10:25:07 GMT X-Page-Speed: 1.9.32.1-4238

Regular headers minus the varnish ones.

[oschaaf]

I'm not sure that pagespeed would treat that as a cacheable html response, as it explicitly advertise that.Could you add an expiry header for the static html file? Also, not relevant here, but later on it will be, the downstream cache vcl is set up to only cache responses that are compressed. So request that don't have Accept-Encoding: gzip in their headers will not be treated as cacheable by varnish (block 3b from https://developers.google.com/speed/pagespeed/module/downstream-caching)

[journeytosilius]

Headers with expires

HTTP/1.1 200 OK Server: nginx/1.7.6 Content-Type: text/html Connection: keep-alive Vary: Accept-Encoding Cache-Control: max-age=2592000 Cache-Control: public Pragma: public Date: Thu, 16 Oct 2014 10:44:36 GMT X-Page-Speed: 1.9.32.1-4238

Calling this URL via Varnish

HTTP/1.1 200 OK Server: nginx/1.7.6 Content-Type: text/html Vary: Accept-Encoding Pragma: public Date: Thu, 16 Oct 2014 10:46:13 GMT X-Page-Speed: 1.9.32.1-4238 Cache-Control: no-cache, max-age=0 X-Varnish: 493363 Age: 0 Via: 1.1 varnish-v4 X-Cache: MISS Connection: keep-alive

[oschaaf]

I'd expect a Content-Encoding: gzip header in the second response? Without 'Accept-Encoding: gzip', the request wil not be treated as cacheable by the vcl. Or is nginx not setup to gzip responses? That would explain this response header in that case, and you could disregard this comment.

Here's a VCL that I think should be working: https://code.google.com/p/modpagespeed/source/browse/trunk/src/install/debug_conf.v4.vcl You'd have to change the backend and ACL, but perhaps you can compare it to what you have now.

[journeytosilius]

Yes Otto, Nginx is setup to gzip everything. As I am calling the URL via curl, it does not show up but If I do it with a browser it does has the gzip header :

Accept-RangesbytesAge0Cache-Controlno-cache, max-age=0Connectionkeep-alive Content-EncodinggzipContent-Length1768Content-Typetext/htmlDateFri, 17 Oct 2014 06:14:14 GMTPragmapublicServernginx/1.7.6VaryAccept-EncodingVia1.1 varnish-v4X-CacheMISSX-Page-Speed1.9.32.1-4238X-Varnish473045

I will try testing that .vlc and report back

2014-10-16 13:10 GMT+02:00 Otto van der Schaaf notifications@github.com:

I'd expect a Content-Encoding: gzip header in the second response? Without 'Accept-Encoding: gzip', the request wil not be treated as cacheable by the vcl. Or is nginx not setup to gzip responses? That would explain this response header in that case, and you could disregard this comment.

Here's a VCL that I think should be working: https://code.google.com/p/modpagespeed/source/browse/trunk/src/install/debug_conf.v4.vcl You'd have to change the backend and VCL, but perhaps you can compare it to what you have now.

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-59346181 .

[journeytosilius]

The vcl works, pagespeed serves optimized pages but still does not hit the cache, what would be the next approach to debug it ?

Headers for the plain HTML file now :

Accept-Ranges bytes Age 0 Cache-Control no-cache, max-age=0 Connection keep-alive Content-Encoding gzip Content-Length 1766 Content-Type text/html Date Fri, 17 Oct 2014 06:55:47 GMT PS-CapabilityList ll,ii,dj: Pragma public Server nginx/1.7.6 Vary Accept-Encoding Via 1.1 varnish-v4 X-Cache MISS X-Page-Speed 1.9.32.1-4238 X-Varnish 426016

By the way, just for reference : Relevant docs: wiki.nginx.org/HttpGzipModule#gzip_types Enables compression for additional MIME-types besides "text/html". "text/html" is always compressed

[oschaaf]

@chromafunk that is surprising. To debug, I would add a few std.syslog calls to the vcl to trace what happens. I'll test the v4 config later today to see if I can confirm this.

[journeytosilius]

Hi Otto, with this .vcl varnish works, but no pagespeed configured. I will try to adapt the pagespeed conf to it and report changes. The Wordpress part is irrelevant to determine if it works or not, because it just makes varnish avoid some cookies and to not cache some areas. So it seems the base config works, I will proceed to adapt the pagespeed part and report back. Any thoughts about it ?

https://github.com/nicolargo/varnish-nginx-wordpress/blob/master/varnish/varnish4-wordpress

[oschaaf]

@chromafunk now that you have varnish working with downstream caching, do you have any feedback on the v4 configuration?

[journeytosilius]

I am close, but not yet.

I am using this config : https://code.google.com/p/modpagespeed/source/browse/trunk/src/install/debug_conf.v4.vcl

And this pagespeed config : https://gist.github.com/chromafunk/5d986063c9036249760e

I am getting consistent hits to the Varnish cache, and pagespeed returns the fully optimized pages. BUT : This is only happening when I do a test from http://gtmetrix.com or http://www.webpagetest.org

When I test the page on a browser with emptied local cache, it does not hit. I have tried on several scenarios, like on other computers with empty local caches in other parts of the world, different countries and the same happens. Do you think it has something to do with the user agent code on the .vcl file ?

It's weird that it only hits from a request made by those services. Otherwise I just get the optimized pages but It bypasses the varnish cache. After debugging this, the next step would be to make it hit consistenty on repeat views unless the user is logged in, but that is a wordpress specific thing. First I need to know why it only hits properly from those services and not from a regular browser on the first visit, non-cached request.

By the way, sharding / paralelization works perfect now with 4 CNAMES.

[oschaaf]

Re: Do you think it has something to do with the user agent code on the .vcl file ? Did you clean out any cookies as well on those browers?

Great to hear sharding is working well now!

[journeytosilius]

Tested erasing cache and cookies, still happens .... no idea why, any ideas ?

2014-10-22 13:30 GMT+02:00 Otto van der Schaaf notifications@github.com:

Re: Do you think it has something to do with the user agent code on the .vcl file ? Did you clean out any cookies as well on those browers?

Great to hear sharding is working well now!

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-60070825 .

[journeytosilius]

Retracted, it works. Did not work from my computer but from another computer in Europe did.

Ok, so from now I will make it always hit and adapt it to wordpress.

Will return with news as soon as I get it.

2014-10-22 14:37 GMT+02:00 Xavier Hoffman xavier@eo10.net:

Tested erasing cache and cookies, still happens .... no idea why, any ideas ?

2014-10-22 13:30 GMT+02:00 Otto van der Schaaf notifications@github.com:

Re: Do you think it has something to do with the user agent code on the .vcl file ? Did you clean out any cookies as well on those browers?

Great to hear sharding is working well now!

— Reply to this email directly or view it on GitHub https://github.com/pagespeed/ngx_pagespeed/issues/829#issuecomment-60070825 .

[oschaaf]

@chromafunk great, sounds good!

[jeffkaufman]

@chromafunk are you still working on this? Do you have more questions or issues with it?