NHentai Cloudflare breaks downloader/metadata plugins

doublewelp commented 2 years ago

I've been receiving "No matching download handler found for url https://nhentai.net" errors even after installing both plugins. After messing around with the downloader to make it log the response content, I figured it out.

NHentai enabled Cloudflare some time ago and this breaks both NHentai plugins - the requests receive a 503 response with the Cloudflare "please wait" screen. I think this could be improved in two ways:

the logging output for plugins should be more specific: HTTP error codes and messages would help diagnose issues;
I'm not familiar enough with Python or Cloudflare to figure out a proper fix, but I figure it should be done at the hpx.command.SingleGETRequest().request level. Some way to trick Cloudflare into believing that it's a real browser doing the interaction?

I've attached both of the edited plugins for anyone who encounters the same problem. This isn't really a fix since the cf_clearance cookie will eventually expire and this forces you to continue editing the plugins. @twiddli , thank you for making this, it's a great piece of software and nothing I've used so far is even close.

nh plugins.zip

Rooba commented 2 years ago

assuming you are running this on a local ip, you would want to occasionally visit any page and then cloudflare will whitelist your connection for some time. if it is remote, you could throw a socks proxy onto the remote machine, apply the socks proxy to a tab in your browser, and visit a page. this is from what i gather at least, i could be wrong but this is at least what i have deduced has fixed it for me

doublewelp commented 2 years ago

I did try a socks proxy, it's a remote machine. I don't know if it's some sort of heuristic being used by Cloudflare, but from my testing it was very stringent - nothing happened with just the proxy visit, and adding the cf_clearance cookie to the plugin wasn't enough. If the user agent used by the plugin didn't match the one from my local browser as well, I'd get bounced by Cloudflare. It's a seedbox, so I guess the IP could be on a suspect list somewhere.

Serph91P commented 2 years ago

I think I have the same problem, but its a local machine. Error Mesaage says:

[---INFO---]--2022-Aug-26 13:56:05-: No download handler returned any results for url https://nhentai.net/g/

doublewelp commented 2 years ago

Have you tried using the edited plugins? I don't think there's a better solution at this point. Other projects like Lanraragi solve this issue by having plugin inputs so you can update your Cloudflare token cookies without hardcoding them inside your plugins, but happypandax looks abandoned at this point.

happypandax / plugins

NHentai Cloudflare breaks downloader/metadata plugins #48