search

nexcess / magento-turpentine

A Varnish extension for Magento.
GNU General Public License v2.0
519 stars 253 forks source link

Crawler stops working randomly #1144

Open R0n1n-77 opened 8 years ago

R0n1n-77 commented 8 years ago

Hi, I noticed that crawler stops working randomly. Flushing Magento cache and restarting Varnish fixes it, but issues starts occurring again after half an hour or hour. In a turpentine log I get:

2016-03-23T10:23:11+00:00 WARN (4): TURPENTINE: Error crawling URL (http://www-test.xxx.xx/xxx/xxx): Unable to read response, or response is empty

In a Varnish log:

19 Debug        c Write error, retval = -1, len = 1422, errno = Broken pipe

Please advise.

R0n1n-77 commented 8 years ago

In addition, browsing to the hash url of the corresponding request like:

/turpentine/esi/getBlock/method/esi/access/public/ttl/86400/hmac/f19dbb753bd2f245f19a562e9beebd6d3211cf5736da12519b596be44eca3469/data/qoB3aU7bq-oZ3EIZMePYpOvkRXoLLY.nu-Y.2Vi2uUddhFoSh789sDCdy1zicSnfZ1KT8SyHfcapVpnse0OfjHZ7PSgUSEs8oMbLROCGiAXMzbNLU2v8EP1

Returns - "ESI data is not valid". Other logs like, PHP, Apache and Magento do not produce any errors. Please advise.

R0n1n-77 commented 8 years ago

~~Noticed another thing. If I remove the real server IP from a "Crawler IP Addresses" and leave just 127.0.0.1, then crawler starts working as expected. But then warmer script becomes useless.~~

Actually found out, that crawler starts throwing errors when the warmer script is running. Is there any config that may cause them to conflict?

aricwatson commented 8 years ago

I haven't experienced this one before. It sounds like an issue occurring when Varnish attempts to connect to the back-end. This seems to make it sound like it might be related to having gzip compression set up for nginx.

R0n1n-77 commented 8 years ago

@aricwatson I don't have gzip enabled in Nginx, although traffic to backend is really proxied through it to apache. Gzip in apache is enabled. Adding a document to .request, as suggested in a link you've sent looks like made a change. Crawler is still throwing errors on more then half of the URLs, but URLs are actually being crawled, F>or I'm getting "X-Varnish-Hits 1".

aricwatson commented 8 years ago

Crawler is still throwing errors on more then half of the URLs,

And it's still the broken pipe error?

R0n1n-77 commented 8 years ago

Yes, right. Varnish still returns the broken pipe error.