search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.45k stars 212 forks source link

rule34hentai no longer working #1858

Closed ayoits0913 closed 1 year ago

ayoits0913 commented 4 years ago

That site has always worked great with this program but as of today, I cannot connect to it. It just says there is no result

HawkSoul commented 4 years ago

Hi, I would think that your page number is incorrect. It should be on first page when u enter new search.

On 31 Dec 2019, at 10:38, ayoits0913 notifications@github.com<mailto:notifications@github.com> wrote:

That site has always worked great with this program but as of today, I cannot connect to it. It just says there is no result

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/Bionus/imgbrd-grabber/issues/1858?email_source=notifications&email_token=ANLR6JMB45Z7NR5XCASO5BTQ3MHIVA5CNFSM4KBUDJAKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IDO53TQ, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANLR6JNZ3OOFGWR7EYELWPLQ3MHIVANCNFSM4KBUDJAA.

ayoits0913 commented 4 years ago

It is on the first page. Even after deleting the appdata folder and doing a fresh install, it isn't working.

I can connect to the site fine with my browser, and even test the login info on the program, and it says "success".. yet it still says "No valid source of the site returned results." I've been checking the site on this program every day for months, very strange.

ayoits0913 commented 4 years ago

This is from the log:

[22:40:46.731][Info] Loading results... [22:40:46.731][Info] [rule34hentai.net][Html] Loading page https://rule34hentai.net/post/list/1 [22:40:46.992][Info] [rule34hentai.net][Html] Receiving page https://rule34hentai.net/post/list/1 [22:40:46.992][Error] [rule34hentai.net][Html] Loading error: Error transferring https://rule34hentai.net/post/list/1 - server replied: Shimmie (203) [22:40:46.993][Warning] [rule34hentai.net] Loading using Html failed. Retry using Rss. [22:40:46.993][Info] [rule34hentai.net][Rss] Loading page https://rule34hentai.net/rss/images/1 [22:40:47.176][Info] [rule34hentai.net][Rss] Receiving page https://rule34hentai.net/rss/images/1 [22:40:47.176][Error] [rule34hentai.net][Rss] Loading error: Error transferring https://rule34hentai.net/rss/images/1 - server replied: Shimmie (203) [22:40:47.176][Warning] [rule34hentai.net] No valid source of the site returned result.

HawkSoul commented 4 years ago

Hi First of all Happy new year and best wishes to all of you!

And second I honestly never used that site, but is there a possibility that the https:// giving trouble? Try to change the url to www..... maybe? Good luck.

On 1 Jan 2020, at 04:41, ayoits0913 notifications@github.com<mailto:notifications@github.com> wrote:

This is from the log:

[22:40:46.731][Info] Loading results... [22:40:46.731][Info] [rule34hentai.nethttp://rule34hentai.net][Html] Loading page https://rule34hentai.net/post/list/1 [22:40:46.992][Info] [rule34hentai.nethttp://rule34hentai.net][Html] Receiving page https://rule34hentai.net/post/list/1 [22:40:46.992][Error] [rule34hentai.nethttp://rule34hentai.net][Html] Loading error: Error transferring https://rule34hentai.net/post/list/1 - server replied: Shimmie (203) [22:40:46.993][Warning] [rule34hentai.nethttp://rule34hentai.net] Loading using Html failed. Retry using Rss. [22:40:46.993][Info] [rule34hentai.nethttp://rule34hentai.net][Rss] Loading page https://rule34hentai.net/rss/images/1 [22:40:47.176][Info] [rule34hentai.nethttp://rule34hentai.net][Rss] Receiving page https://rule34hentai.net/rss/images/1 [22:40:47.176][Error] [rule34hentai.nethttp://rule34hentai.net][Rss] Loading error: Error transferring https://rule34hentai.net/rss/images/1 - server replied: Shimmie (203) [22:40:47.176][Warning] [rule34hentai.nethttp://rule34hentai.net] No valid source of the site returned result.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Bionus/imgbrd-grabber/issues/1858?email_source=notifications&email_token=ANLR6JOOGKJOCLUNUTDJSH3Q3QGFVA5CNFSM4KBUDJAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH44RQY#issuecomment-570017987, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANLR6JJVALXJZI3UBMNETNTQ3QGFVANCNFSM4KBUDJAA.

ayoits0913 commented 4 years ago

Happy New Year.

Unfortunately, that made no difference. Not sure what to do.

ayoits0913 commented 4 years ago

If anyone can test the site and let me know if it's just on my end or not, I'd appreciate it. This issue makes no sense to me, especially since I can successfully test the login info and use the site with my browser.

trucman commented 4 years ago

I'm having the same issue for a few days already :/ except the login test will result in failure for me, and it's working when I'm using a browser...

edit : update, I can now login sucessfully but still no result.

HawkSoul commented 4 years ago

I just did the test and this showed up: [image1.jpeg] So i chose Yes and images just loaded fine. The website seem a little different “rule34.xxx.”?

Hopefully that helps.

Cheers

On 3 Jan 2020, at 11:45, trucman notifications@github.com<mailto:notifications@github.com> wrote:

I'm having the same issue for a few days already :/ except the login test will result in failure for me, and it's working when I'm using a browser...

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Bionus/imgbrd-grabber/issues/1858?email_source=notifications&email_token=ANLR6JMV6SY6QPPGQ33KOMTQ34JNLA5CNFSM4KBUDJAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIA3KZA#issuecomment-570537316, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANLR6JNWDRJLKGNQE3DAW5LQ34JNLANCNFSM4KBUDJAA.

ayoits0913 commented 4 years ago

rule34.xxx is an entirely different website. That works fine for me as well. rule34hentai.net you have to add manually (not included in the sources), can you test it?

jkaryan commented 4 years ago

Same issues here, the website is completely inoperable in Grabber. Real shame, it's been one of my main sources.

HawkSoul commented 4 years ago

Same here, and i cant even open it in my browser. Gets stuck on anti-Ddos attack check

On 6 Jan 2020, at 12:08, jkaryan notifications@github.com<mailto:notifications@github.com> wrote:

Same issues here, the website is completely inoperable in Grabber. Real shame, it's been one of my main sources.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Bionus/imgbrd-grabber/issues/1858?email_source=notifications&email_token=ANLR6JNJ4ZQZDACNYMBTZDLQ4MGI7A5CNFSM4KBUDJAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFETSY#issuecomment-571099595, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANLR6JOSETXRHFYEBXZJTELQ4MGI7ANCNFSM4KBUDJAA.

jkaryan commented 4 years ago

Same here, and i cant even open it in my browser. Gets stuck on anti-Ddos attack check

That one seems to have been fixed/come and go. It didn't work for me yesterday, does work for me today.

ayoits0913 commented 4 years ago

I hope Bionus checks this issue out, this is such a great program and that's my favorite site.

CuddleBear92 commented 4 years ago

This is all a CloudFlare issue, admin specifically set this up to stop bots and other downloaders from accessing the site.

Best way to fix this would be a headless browser for example that could approve the gated entry. Hydrus is having the same issues at this moment.

Sad to see the site that has so much exclusive content not mirrored elsewhere. All gated off from download clients, filled with ads and limited tags. Seems like admin struggles with having a grasp on the site.

Hopefully @Bionus can take a look at it and maybe find a good and simple option. The Hydrus guys will check in here from time to time for re-useable results hopefully. (need a full redesign for a headless browser)

Bionus commented 4 years ago

Indeed, the owner of the website set up CloudFlare on their website, which blocks the entry of programs such as Grabber. Usually this can be circumvented by loading the page in a browser then copying the CloudFlare cookies to Grabber (IIRC it's _cfduid), but note that I haven't tested it here (I can't even access the website).

Best way to fix this would be a headless browser for example that could approve the gated entry.

Hopefully @Bionus can take a look at it and maybe find a good and simple option. The Hydrus guys will check in here from time to time for re-useable results hopefully. (need a full redesign for a headless browser)

As you said, the solution is to load the website in a headless browser (same thing for a few websites that for example use React without server-side rendering), then get the results there. But just like Hydrus, that would require a huge re-write in Grabber.

ayoits0913 commented 4 years ago

I just tried adding that cookie, didn't change anything unfortunately.

CuddleBear92 commented 4 years ago

No sadly just the cookie wouldn't help. There is an 1 hour access before re-verifying the access going on now from what i gather from general usage in browser.

Really hate this as R34H.net has so many 3D rendered releases and other paygated releases compared to others and does so with earlier uploads too.

It was a good decent site in terms of pure content and its upload consistency if you enjoy the content uploaded. Sadly the admin, ads, cloudflare and lacking tags detract from the site as a whole.

Sad to see the site get locked down like this as the admin uploads/scraping scripts seems to be on point. Direct from the sources of all the creators at raw quality. Something no other booru does for 3D rendered stuff atleast.

Someone in the Hydrus Discord did link to this though, some have stated that it works well with cfscrape (linked bellow)

This will only be the start sadly, its only a matter of time before more fall like this, Sankaku is prob next with how they thread request timers and other clients.

Relevant links: https://github.com/venomous/cloudscraper https://github.com/Anorov/cloudflare-scrape

SultrySamthepennanceman commented 4 years ago

You know, I just came in to talk about this exact issue after redownloading grabber to a new computer and getting set up again, and then I read the thread. God-damnit, God fucking dammit, probably the second worst shit of the new year I've had to come to.

ayoits0913 commented 4 years ago

You know, I just came in to talk about this exact issue after redownloading grabber to a new computer and getting set up again, and then I read the thread. God-damnit, God fucking dammit, probably the second worst shit of the new year I've had to come to.

I feel you man. It's a big loss. I'm not about to download one image at a time, some tags have thousands of images. This program made it so easy.

Spyridion commented 4 years ago

So just coming back to this thread, I'm having certain queries go through and others not. They just hang with no error. Is this the cloudflare blocking grabber?

CuddleBear92 commented 4 years ago

@Spyridion The admin of the site did setup Cloudflare for its site. On the Hydrus side we have seen the cloudscraper i linked last year have worked a bit. But the dev of that have paygated some features so far, adding the 2captcha api to the parser is prob needed in the end.

We have also seen this mostly be an issue outside of the US, if you VPN to the US then you SHOULD be good aswell.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

If this issue is about a bug that still happens in the latest version, or a suggestion that is still relevant, feel free to comment on it and the maintainers will have another look, they might have missed it!

Thank you!