Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.55k stars 217 forks source link

[Bug][Nightly] List of sites that work, don't work, and partially work (data for latest nightly and 7.10.0 for windows x64 portable) #3052

Open suminohana opened 1 year ago

suminohana commented 1 year ago

Bug description

I was going to report a simple bug initially, but ended up doing quite a long research here.

I couldn't test the latest release due to qt5Print dll issues, which is why I'm using nightly and 7.10.0.

Here's a list of sites that work, don't work, and partially work (have issues with search function):


On Nightly

Working:

derpibooru e-hentai, e621 hentai-cosplays.com (keyword search) kemono.party konachan rule34.us tentaclerape.net deviantart (it crashed on the first search but worked normally after) zerochan

-

Search not functional:

rule34.xxx (all searches return the same results) danbooru gelbooru pawoo.net (says "server offline" or "all images filtered" on search) rule34.paheal.net (says "no results") safebooru.org (all searches return the same results) wallhaven.cc (nonsensical results) artstation (says "server offline" on search) yande.re (some results make no sense, eg. when you search for black_hair or other hair colours, they all show the same 15 unrelated results)

-

Not Working:

anime-pictures.net (says "no result") Twitter (I assume I'd have to login) booru.io (nothing shows up) hijiribe.donmai.us (says "server is offline") nhentai (says "server is offline") nijie.info (works once and then dies) sonohara.donmai.us (says "server offline") newgrounds (says "no result") pixiv (nothing shows up) reddit (says "no result" and causes repeated crashes if you try to search something)

All searches were done with default settings.


On 7.10.0

Working:

Rule34.xxx Behoimo Booru.io Gelbooru derpibooru e-hentai e621 konachan rule34.us safebooru tentaclerape.net

Search not functional:

pawoo.net rule34.paheal.net wallhaven.cc (results are not consistent with tags) artstation (says "no results") reddit (it kinda works but not really useful) yande.re (some results make no sense, eg. when you search for black_hair or other hair colours, they all show the same 15 unrelated results)

Not Working:

anime-pictures.net (says "no result") Twitter (I assume I'd have to login) Danbooru (says "server is offline") hijiribe.donmai.us (says "server is offline") nhentai (says "server is offline") nijie.info (worked once and then died) slushe (says "no result") sonohara.donmai.us (same issue as other donmai domains) pixiv (nothing shows up)

All searches were done with default settings.


These sites have AWFUL tagging and are basically pointless: http://behoimi.org/, https://derpibooru.org/,

Steps to reproduce

download portable of the 2 specified versions for windows x64.

Expected behavior

for it to work normally

Context

the reddit crashes produced no error messages.

System information

suminohana commented 1 year ago

sites that work on both:

derpibooru e-hentai e621 konachan rule34.us tentaclerape.net

-

sites that partially work on both:

pawoo.net rule34.paheal.net wallhaven.cc artstation yande.re

-

sites that don't work on both:

anime-pictures.net twitter hijiribe.donmai.us nhentai nijie.info sonohara.donmai.us pixiv

Bionus commented 1 year ago

Hello πŸ‘‹

Thanks for the very complete report! It's a bit long so let me take each point one at a time.


First, indeed it seems that search on Danbooru, Gelbooru, and sources based on those were broken (the search input wasn't properly sent in requests, so all searches were the same) in Nightly. Actually, sources having this new dropdown here were broken: image I just pushed a fix, so those four sources you listed should now be fixed:

Some other sources simply do not support searching at the moment, or require to login first:

Some sources even require login to perform anything:

While for some other sources, it seems that search is broken directly on their website:

Two sources seem to indeed be totally broken though, good catch!

And finally, for some sources I can't reproduce:


In any case thanks for the heads up, I'll do some more digging for broken sources, and try to make the actual error messages clearer, as simply saying "server offline" is definitely confusing for sources that actually require to login instead.

suminohana commented 1 year ago

@Bionus so, as it turns out, rule34.paheal.net DOES work, it's just that the site is a piece of crap and their tagging system is terrible. I'll add it to the pointless sites list.

I tested booru.io again and still nothing shows up on search. it says "Page 1 of ? (50 of ?)".

Regarding the pointless sites list, I think you should really consider removing support for them, as it will only be a burden and these sites are just so bad they're hardly useful.

suminohana commented 1 year ago

Pointless sites list: http://behoimi.org/, https://derpibooru.org/, https://rule34.paheal.net

suminohana commented 1 year ago

I'll test the new nightly right now

ps: nevermind the nightly is still from 1 month ago

Bionus commented 1 year ago

Yup, nightly build seems to be broken: https://github.com/Bionus/imgbrd-grabber/actions/workflows/build.yml?query=branch%3Adevelop I'll check this out, sorry about that πŸ€”

Bionus commented 1 year ago

Regarding the pointless sites list, I think you should really consider removing support for them, as it will only be a burden and these sites are just so bad they're hardly useful.

If they work and they don't cost much to keep supporting, I don't really see a point to remove them. I agree some of them have pretty bad tagging, but they also have content that might not be as easily found as other places, so interested users would still want to be able to access them. I'd say "useless" is too relative πŸ˜„

uvhat commented 1 year ago

some advices: For the nhentai you should use fresh cookies and exact same user agent from your browser instance that you used for getting the fresh cookies. And all works. This is working for another sources with cloudflare.

Use any cookie manager extension to copy cookies (cf_clearance, csrftoken, cf_chl_2 and other) and set User-Agent exact same to you browser. After cookies expired - refresh them again.

suminohana commented 1 year ago

Report on sites previously with issues:

1 - anime-pictures.net: it now works, but the search results can be a bit odd sometimes, like when you search for red_hair or brownhair. I think it considers to be equal to a space and reads it as 2 separate tags or something. In any case it will sometimes ignore the fact there is a _hair and display red_eyes pictures (for red_hair), and brown_eyes for brown_hair. Apart from that, no other issues

2 - rule34.xxx: seems to be working as intended

3 - booru.io: is still not working on my end

4 - danbooru: seems to be working as intended

5 - gelbooru: seems to be working as intended

6 - hijiribe.donmai.us: still doesn't work

7 - pawoo.net: still no message saying I have to login

8 - safebooru: seems to be working normally

9 - sonohara.donmai.us: still not working

10 - wallhaven.cc: the problem seems to be on the underscore ( _ ). even though tags have underscores, if you try to use them it will return no results. this also happens on their website.

11 - artstation: search doesn't work at all

12 - newgrounds: still doesn't work

13 - reddit: still crashing on search


Report on other sites:

1 - derpibooru: can't use underscore ( _ ) on search. you need to use a space and use commas ( , ) to separate tags.

2 - e621: searches causing crashes, similarly to reddit

GiovanH commented 1 year ago

Can confirm Newgrounds is still offline. Logs indicate the request isn't passing parameters:

[00:46:25.900][Info] Loading results...
[00:46:25.903][Info] [www.newgrounds.com/art][Json] Loading page http://www.newgrounds.com/art/browse?interval=all&sort=date&genre=0&artist-type=all&offset=0
[00:46:27.061][Info] [www.newgrounds.com/art][Json] Receiving page http://www.newgrounds.com/art/browse?interval=all&sort=date&genre=0&artist-type=all&offset=0
[00:46:27.063][Error] [www.newgrounds.com/art][Json] Loading error: Error transferring http://www.newgrounds.com/art/browse?interval=all&sort=date&genre=0&artist-type=all&offset=0 - server replied: Bad Request (302)
[00:46:27.064][Warning] [www.newgrounds.com/art] No valid source of the site returned result.

I would expect something like https://www.newgrounds.com/search/conduct/art?suitabilities=e&c=4&terms=pomni

Tested using 7.11.2