search

Flameish / Novel-Grabber

Novel-Grabber can download novels from pretty much any webnovel and lightnovel site.
MIT License
483 stars 63 forks source link

[BUG] Cannot bypass cloudflare #135

Open jsanchezflores opened 3 years ago

jsanchezflores commented 3 years ago

Describe the bug [ERROR]Service Unavailable! (503)

Cannot download from foxaholic because novel-grabber/in-app browser Chrome (tried others also) cannot bypass cloudflare. I can access Foxaholic just fine with my regular browser (Chrome).

Expected behavior It seems to have happened recently because I was able to download with no issues before.

Platform Windows, Java 8

Logs

[ERROR]Could not establish connection to SMTP Server. Check email settings and restart.
[INFO]Checking World?s Fastest Level up! ~as a Result of the Awakening of My Incompetence Skill [Transfer Within Dungeon], I Became the Strongest Without Being Restricted by the Rules of the Dungeon.
[ERROR]Service Unavailable! (503)
[INFO]Checking A Banished Failure Survives in the Borderland and Becomes an S-Rank Exorcist
[ERROR]Service Unavailable! (503)
[INFO]Checking Disciple of Immortal
[ERROR]Service Unavailable! (503)
[INFO]Checking I Became the Strongest With The Failure Frame?Abnormal State Skill?As I Devastated Everything
[ERROR]Service Unavailable! (503)
Flameish commented 3 years ago

Nothing I can do about that, I searched long for a solution but haven't found one yet. You can try at a later time again, Cloudflare is often only temporary.

Flameish commented 3 years ago

Keeping it open for visibility.

lunevale commented 3 years ago

Just wondering, would NG launcher work if I used something like https://github.com/VeNoMouS/cloudscraper to bypass the cloudflare on foxaholic first?

jsanchezflores commented 3 years ago

Foxaholic removed cloudflare protection for the time being. It's been working for me fine since then.

lunevale commented 3 years ago

@jsanchezflores Thanks for letting me know! It's working for me now too. How do you know when they've removed cloudflare protection though?

jsanchezflores commented 3 years ago

I use the website often and the cloudflare protection wasn't checking with my browser anymore, also remember to add wait time of at least 10 (10000) sec when downloading novels. This is one of the reasons why they even add protection in the first place to prevent this.

thornt5748 commented 3 years ago

Currently trying to use it now, but receiving an error message [ERROR] Could not connect to webpage!

Steps take: Launch NG-Launcher.jar Entered Novel URL: https://www.foxaholic.com/novel/i-was-caught-up-in-a-hero-summoning-but-that-world-is-at-peace/ Clicked "Check availability of Novel"

frykauf commented 3 years ago

Just wondering, would NG launcher work if I used something like https://github.com/VeNoMouS/cloudscraper to bypass the cloudflare on foxaholic first?

Or what Jdownloader does, just making the automation launch the browser with specified/default profile, that should include all the cookies for novelupdates and other sites as long as you completed the captcha once before and it saves to that browser profile’s cookies.

Flameish commented 3 years ago

Cloudflare is pretty good at detecting automated browser activity. Cookies won't matter in that case.

frykauf commented 3 years ago

Then maybe separate hCaptcha manual solver either in set browser or in app pop up window? Some downloaders do that. JDownloader does it through the separate manual solver & browser extension to pass the solve to Jdownloader & some downloaders do it via pop up browser window through internal headless browser - there you manually solve the captcha without browser/browser extension.

I do wonder if by this point if it wouldn’t be easier to make Novelupdates Jdownloader plugin, that passes the data to Novel Grabber.