search

scrapinghub / splash

Lightweight, scriptable browser as a service with an HTTP API
BSD 3-Clause "New" or "Revised" License
4.09k stars 513 forks source link

Using HA Proxy (Aquarium), still getting lot of 504/503 errors. #750

Closed nehakansal closed 4 years ago

nehakansal commented 6 years ago

I am using HA Proxy (via Aquarium) and I also increased the timeout setting to 300 just to have a higher value for testing. I get a bunch of 504 timeout errors (in the docker logs) on urls that I am pretty sure are quick to crawl. And I also get 503s service unavailable errors (in the crawl logs - running Splash for Undercrawler - https://github.com/TeamHG-Memex/undercrawler). HA Proxy stats page shows that it couldnt reach Splash for a few minutes and I don't see splash-going-down related errors. Each crawl is different, doesnt produce similar number of errors each time. I would think that since HA Proxy manages the request queue, 503/504 should not ideally happen. What could cause these errors?

nehakansal commented 6 years ago

It seems I was getting these, a majority of them if not all, due to running out of memory. Please see issue #757

Gallaecio commented 4 years ago

Closing this in favor of https://github.com/scrapinghub/splash/issues/757