BFI runs being blocked sometimes by cloudflare

alistairjcbrown commented 2 weeks ago

Hypothesis: Lots of regular traffic from the same IP at times when there's little other traffic (can't imagine their site is busy at 4am)

Proposals:

Increase the number of overall retries -- retrying at that stage is cheap, as everything that was gather before the failure has been cached. Bump the number up to 20?
Increase the wait before trying again -- make it 30 seconds?
Retries within the code -- is it worth having code that just closes the browser, reopens it again to see if it now gets through?
Random delays -- would it be less likely to be caught if there were some delays sprinkled between requests?
Move the mouse and click the human button -- does that work?
Others must also have solved this problem, e.g. https://www.npmjs.com/package/puppeteer-extra-plugin-stealth, https://www.zenrows.com/blog/puppeteer-cloudflare-bypass#bypass-cloudflare-with-zenrows

alistairjcbrown commented 3 days ago

Retries increased, wait before retry increased, and stealth plugin added. There was a successful run with this setup -- https://github.com/alistairjcbrown/hackney-cinema-calendar/actions/runs/11900772403/job/33162316336 Let's see what happens tomorrow morning.

alistairjcbrown commented 3 days ago

Will keep this open for a few more days to see if it's consistent

alistairjcbrown commented 22 hours ago

It's run well for the last few days -- closing this out, we can re-open if failures appear in the future

alistairjcbrown / hackney-cinema-calendar