vogler
commented
2 years ago Proxies to bypass Cloudflare protection also have problems with hcaptcha:
- https://github.com/FlareSolverr/FlareSolverr/issues/128
- https://github.com/NoahCardoza/CloudProxy#captcha-solvers
- CaptchaHarvester collects for manual self-solving instead of paying for 2captcha etc.
- hcaptcha-solver randomly selects images until solved
There are several browser extensions but extensions can not be used in headless mode: https://playwright.dev/docs/chrome-extensions
Free captcha solvers:
- https://github.com/dessant/buster browser extension that uses speech recognition to solve reCAPTCHA audio challenges
- no support for hcaptcha: https://github.com/dessant/buster/issues/319
- https://www.npmjs.com/package/puppeteer-hcaptcha uses TensorFlow's image recognition (unreliable?)
- https://www.nopecha.com/ - chrome extension seems to be free and work well
- https://github.com/Sqaaakoi/noptcha-patch - above, but without affiliate tracking code injection
- https://github.com/shimuldn/hCaptchaSolverApi - says 1000/month for free?
- https://github.com/privacypass/challenge-bypass-extension - allows to solve challenges in advance; "Privacy Pass is currently supported by Cloudflare to allow users to redeem validly signed tokens instead of completing CAPTCHA solutions."
General info, detecting headless mode, anti-scraping:
- https://stackoverflow.com/questions/55493536/how-to-deal-with-the-captcha-when-doing-web-scraping-in-puppeteer/55500565#55500565
- https://intoli.com/blog/making-chrome-headless-undetectable/ - example for
chrome-remote-interface - https://help.apify.com/en/articles/1961361-several-tips-on-how-to-bypass-website-anti-scraping-protections
- https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ - frameworks will always be detected, use normal browser; also has examples for simulating human mouse movement and typing
- https://stackoverflow.com/questions/62751377/bypass-cloudflare-with-puppeteer#comment117761360_62751377
- "The only workaround that I could make work involved using a non-headless browser over the same IP address and snapshotting the cookie info from the site (which included the CloudFlare cookies) and then using those in the cookie jar for my Puppeteer and Perl scripts. It's less than ideal because it's not fully-automated, but it did work for me. Those cookies appear to be good for at least 3 months, so every 3 months I have to just manually get/set them again."
- https://github.com/berstend/puppeteer-extra/issues/614 - Non-headless via docker + VNC
- https://github.com/berstend/puppeteer-extra/issues/609 - control Chrome through debug port
browserWSEndpoint
Currently the script runs not in headless mode, i.e., the automated browser is visible. Comment out this line to not show the browser: https://github.com/vogler/epicgames-claimer/blob/fd3f947223733b187c5e1903be1209f8fd3b618c/main.stealth.js#L28
Running in headless mode resulted in the 20s timeout and the game not being claimed. https://github.com/vogler/epicgames-claimer/blob/fd3f947223733b187c5e1903be1209f8fd3b618c/main.stealth.js#L124 Running in headful mode afterwards successfully claimed the game.
Assumption is that the captcha is shown in headless mode, despite the stealth plugin.