search

nexcess / magento-turpentine

A Varnish extension for Magento.
GNU General Public License v2.0
519 stars 253 forks source link

Crawler doesn't warmup the cache #255

Closed tiennd79 closed 10 years ago

tiennd79 commented 11 years ago

Hi, I have setup the extension and it works great with ESI. My website has two stores and using the browsers to access, it works just great, each stores are cached and all sub-sequent accesses are very fast. After turning on the crawler and setup cronjob to run every 1 hour, in log file I see crawler accesses the url and there is no error reported. But when I copied and pasted those url to browser and use inspector to see header, the page speed is slow and Varnish cache hit is 0. Do you know why the cache was not warmup?

Thank you

aheadley commented 11 years ago

Do you have the Normalize User Agent option enabled? Neither the cache warming script or the crawler pretend to be desktop browsers so with that option enabled the cache will not be warmed for regular users, only bots like GoogleBot.

tiennd79 commented 11 years ago

Hello, My Normalize User Agent is Disabled, that is the right setting, correct?

Today I tried using warm-cache.sh, the tool takes sitemap.xml and crawls all url fine but when copy and paste the url which was just crawled by siege and access from Chrome or Firefox the X-Varnish-Hits is still 0. I also tried to add -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/17.0 Firefox/17.0" to siege command in warm-cache.sh but still didn't help. Thank you for your help.

aheadley commented 11 years ago

My Normalize User Agent is Disabled, that is the right setting, correct?

Yes, it sounds like something else is going on here. Are you sure the domain in the URLs you're testing exactly matches what the crawler is hitting (e.g. www.example.com vs example.com)?

tiennd79 commented 11 years ago

Yes the domain is correct. I am checking the generated VCL to see if varnish caches per user agent.

drk1 commented 7 years ago

Hi, I've exactly the same problem. Everything seems working fine without error, but when I try to reach the urls with my browser after the cache warmer exec, there is no effect (needs to reload to see fast loading).

@tiennd79 : did you found a solution?