504 Gateway Time-out

[odmaaa]

Hello, I am crawling a website with 10K contents, when I crawl first it's all response 200, everything is ok, but after few minutes 504 Gateway Time-out appears and after 3 times retrying scrapy give up retrying. I set :

    'CONCURRENT_REQUESTS':10,
    'HTTPCACHE_ENABLED':True,
    'DOWNLOAD_DELAY':5,
    'CONCURRENT_REQUESTS_PER_IP':10,

and endpoint is render.html

'splash' : {
    'endpoint' : 'render.html',
    'args' : {'wait':1},
}

I am using : scrapy version: 1.0.3 python：2.7 *docker server

How can I optimize my crawler ? and avoid 504 error?

[kmike]

Hey @omkaaa,

Please check http://splash.readthedocs.org/en/stable/faq.html - does it help?

[odmaaa]

Hi @kmike ,

Yes it helped thank you,I disabled the image and set time-out to 720, all worked great. Thank you

[kmike]

@omkaaa glad to hear that!

[yeszao]

Follow @omkaaa, I change the args to:

yield SplashRequest(url, self.parse, args={'wait': 0.5, 'viewport': '1024x2480', 'timeout':90, 'images': 0}

It works!

Besides, some website would very quick when you using curl or Browser, but very slow in splash, because splash cannot download some resources currectly.

These can also come across with 504 Gateway Time-out. The right way is stop the slow resource download. in Splash, you can set resource_timeout in args:

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url,
                                self.parse,
                                args={'wait': 0.5, 'viewport': '1024x2480', 'timeout': 90, 'images': 0, 'resource_timeout': 10},
                                )

[kamrankausar]

Thanks @yeszao it works