search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.47k stars 214 forks source link

Integration with FlareSolverr/other Cloudflare solvers #2649

Open aaabbbccc16 opened 2 years ago

aaabbbccc16 commented 2 years ago

Is your feature request related to a problem? Please describe

At least 2 boorus that I know (furry.booru.org and rule34hentai.net) have the new Cloudflare challenge that can only be solved with an actual browser. It's not much, but more may appear in the future, so it would be reasonable to start considering solutions for this problem. In addition, whenever grabber encounters this protection, it displays a generic "Possible reasons: server offline" message that is not very helpful to understanding what the actual source of the problem is and how the user can deal with it. Currently, to bypass the new challenge it is necessary to open the site in a browser, wait for it to solve the challenge and copy over the User-Agent and cf_clearance cookie from the developer console into the source's settings, which will allow the user to browse the site for a day or so.

Describe the solution you'd like

Jackett deals with the same problem by using FlareSoverr proxy that uses a headless browser to solve the captcha. Although, from the README's wording it sounds like FlareSolverr proxies all the incoming requests through the browser instead of just passing the clearance cookie back, which would be a bit of a problem for the amount of requests Grabber normally makes. I would very much like to see this or similar automated solution being integrated into Grabber.

Describe alternatives you've considered

It that is not feasible, I would like to see some semblance of semi-automation for manually retrieving the clearance cookie from the browser, or at least a separate error message for Cloudflare challenge related errors with information on how to manually retrieve the cookie from the site.

aaabbbccc16 commented 2 years ago

Other cloudflare scrapers mentioned here: https://github.com/Bionus/imgbrd-grabber/issues/1858#issuecomment-573377447

Bionus commented 2 years ago

Rather than implementing a solver, isn't it more reliable to have a built-in browser open so that the user can "solve" the Cloudflare challenge?

aaabbbccc16 commented 2 years ago

Well, really, I meant "any way of (semi-)automatically retrieving the clearance cookie" when I said "solver".