Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
I crawl a web site which network is very bad, I have to refresh 5 times or even more to get a normal response. I tried to configure retry_request_errors, but found that only retry 3 time.
imtiazwazir
commented
4 years ago
I crawl a web site which network is very bad, I have to refresh 5 times or even more to get a normal response. I tried to configure
retry_request_errors, but found that only retry 3 time.https://github.com/vifreefly/kimuraframework/blob/96fe695fe4cbcf1182389e2f867debeca746022d/lib/kimurai/capybara_ext/session.rb#L11
It would be great to improve on Supporting custom
max_retries.