Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.55k stars 216 forks source link

gelbooru data pulls are empty #2757

Closed Arucard707 closed 2 years ago

Arucard707 commented 2 years ago

gelbooru shows content is available and Grabber shows gelbooru recognizes content is available within the Tags screen, but the image previews (Downloads previews ) show there is no result and therefore there is no option to download. I have checked a few other sites within sources and they provide results and previews as intended. I have tried a variety of searches within gelbooru and they are all the same results. This started a couple days ago, most likely when 7.91 was released and somehow interfered with 7.90, after update to 7.91 it still occurs.

I have provided examples with the same search as an example. Please let me know if any additional data is needed.

Screen Shot 2022-07-17 at 11 43 44 PM Screen Shot 2022-07-17 at 11 44 02 PM

xDaSkLox commented 2 years ago

Can confirm the same. Happened right after updating and have been reinstalling and adjusting settings to try and fix it with no success.

HanatakeYurii commented 2 years ago

Same her. The log gives me: [12:23:15.364][Warning] [gelbooru.com][Xml] Search error: We have exceeded maximum queries per day for anonymous usage. Yukkuri Shiteitte ne! :< [12:23:15.366][Warning] [gelbooru.com] Loading using Xml failed. Retry using Html.

It reacts as if I was not logged in. When I test my credentials, I get 'Success', but that doesn't change anything.

It it a Grabber defect or should we change something?

Gelbooru is working fine on my web browser.

ikhlaz commented 2 years ago

It's not just Gelbooru. Rule34.paheal and Rule34.xxx turns up empty too.

lx30011 commented 2 years ago

Same issue here. My credentials are saved but I also get

[15:42:14.754][Warning] [gelbooru.com][Xml] Search error: We have exceeded maximum queries per day for anonymous usage. Yukkuri Shiteitte ne! :<
[15:42:14.754][Warning] [gelbooru.com] Loading using Xml failed. Retry using Html.
Arucard707 commented 2 years ago

I kept getting the same thought around max amount of downloads, since the queries were working then they stopped. Is this a gelbooru or an incorrect user logging issue from Grabber. Could Grabber automatically clear and restart that user every X amount of times or each time for that matter ?

Bionus commented 2 years ago

Does this also happen on 7.9.0? Or is it 7.9.1 specific? 🤔

CH4SSEUR commented 2 years ago

Does this also happen on 7.9.0? Or is it 7.9.1 specific? 🤔

same problem for me happen on 7.9.1 tbh i thought they updated their api or something

edit: same problem on previous version (7.9.0) before this problem Gelbooru seemed getting very slow to load

Hojo-Norem commented 2 years ago

I'm getting this too. I can't remember exactly what version I was on (Probably the one just before 7.9.0), but I was getting the problem with that version. It didn't feel like it started instantly to me either, more like Gelbooru access slowed down something fierce until it just up and quit with 'No Results'.

I upgraded to current to see if that would fix the issue... no success.

I will say that I don't log into Gelbooru, I pretty much never have.

Triramama commented 2 years ago

no images load at all for me, the search returns "no result" with any query, what's weird is that i can use login function as always, i believe this issue is present on every version of the app because before updating to the latest release today i had been using a build from like half a year ago, if it works for you please report what version you are on. so what's going on, any takes, Bionus? edit: yep, tried installing various builds from last year, the same stuff

HanatakeYurii commented 2 years ago

@ikhlaz

It's not just Gelbooru. Rule34.paheal and Rule34.xxx turns up empty too.

Those are working for me, I only have a problem with Gelbooru. Maybe try a clean install if you have problems with other sources.

I have to add after the error message in the logs which seems linked ti XML query, Grabber tries HTML but returns no error and no images so far.

Arucard707 commented 2 years ago

I was checking in older problem tickets and it seems like it kicks things out from time to time , but what's odd to me is that the issue states "We have exceeded maximum queries per day for anonymous usage" which would imply that after 24-48 hours of non usage it should allow normal functions, but it doesn't seem to be resetting even after the program has been off for X amount of time ... curious what cache isn't being flushed or what timer isn't being reset

Atulin commented 2 years ago
[04:46:03.472][Warning] [gelbooru.com][Xml] Search error: We have exceeded maximum queries per day for anonymous usage. Yukkuri Shiteitte ne! :<
[04:46:03.472][Warning] [gelbooru.com] Loading using Xml failed. Retry using Html.

from what I figured out, Gelbooru now requires the API key and user ID on every request. Username and password seem to be not quite enough

Arucard707 commented 2 years ago

Can we either extend the limit of the API Requests manually or is that something within Grabber that is fixed? also what login will it need if the username and password isn't enough ?

Atulin commented 2 years ago

The workaround currently seems to be switching login method to URL, and providing the API key and user ID you can find in Gelbooru options. It does need an account.

Arucard707 commented 2 years ago

So I got it working with the API access at the bottom of the options on Gelbooru but who know how long that will hold. Hopefully the api holds without issue

Triramama commented 2 years ago

i would like it pointed out if im missing something here. I entered the numerical user id like it appears on gelbooru , and entered the api line and nothing still works for me Edit: So I created a new account and copied credentials from there, still no luck, do you know if i need some special privileges on an account for this?
this is what the log gives sometimes at my searches btw: [Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [Warning] [Qt][default] QSqlDatabasePrivate::addDatabase: duplicate connection name 'Tag database - C:/Users/*/AppData/Local/Bionus/Grabber/sites/Gelbooru (0.2)/gelbooru.com/tags.db', old connection removed.. [12:39:50.224][Warning] [Qt][default] QIODevice::read (QNetworkReplyHttpImpl): device not open it parses my key, but results don't show up, it even shows the number of pages in the log but images are at zero I can only access gelbooru from vpn and proxy if that matters

Dprasni commented 2 years ago

So I got it working with the API access at the bottom of the options on Gelbooru but who know how long that will hold. Hopefully the api holds without issue

Could you add more detail here ? I'm also trying it but i am not having luck, it might help if you share how you did it or what you edited where, blurred pics could also help :)

Atulin commented 2 years ago

@Dprasni Here's where to find the credentials: image And here's where to use them and how to set up the login: image

Arucard707 commented 2 years ago

Hey @Dprasni @Atulin beat me to it lol, Butonce you sign into Gelbooru website , and in the settings page at the very bottom, you will see a code like in the first picture, you have to break down the whole code as Atulin shows very well. You will have to input those specific characters in both areas to work. Grabbers sources (Pic 2) is in the lower left hand corner of the search page when you are looking for something . Then the rest is save and restart grabber for the credentials to take effect. (at least that's what I did )

HanatakeYurii commented 2 years ago

I don't get why but Gelbooru is working again on my side without doing anything. Seems something is happening on their end.

Hojo-Norem commented 2 years ago

Same here. Still not logged in. Now working.

That being said, I did notice that while this was happening, yande.re wasn't updating as in there hadn't been any new posts. As soon as Gelbooru started working, so did new posts start appearing on yande.re.

Probably just a coincidence.

HanatakeYurii commented 2 years ago

Probably just a coincidence.

Maybe not, I know multiple sources are replicating their content by fetching them on other sources, like Gelbooru is fetching from Danbooru. Gelbooru is one of the most 'complete' image board, you can expect many others depending on it. It's then fairly possible that Gelbooru was also unavailable for Yande.re .

Arucard707 commented 2 years ago

My best guess is in the issue itself: "We have exceeded maximum queries per day for anonymous usage" The per day usage must have reset and allowed access again ... now where it is and where it was referencing ie Danboru or donna... is unknown but possibly relevant. My best guess is it will happen again eventually but the time when it does will be sporadic

Arucard707 commented 2 years ago

or it's limit for direct api using?

I think it's a built in limit to Gelbooru which it's working for me now but anything over 1000 entries and now it's only downloading 100 then it drops 1000 from the count and will even go in the negative

xDaSkLox commented 2 years ago

I also just noticed the total images per page not being the correct number. I have mine set to 1000, but it only shows 100 per page. In some cases, it will even show an incorrect page number count (e.g. page 3 of 1).

brazenvoid commented 2 years ago

Gelbooru has limited access to anonymous usage, it is a two way restriction:

  1. Maximum downloads per search is limited to 100.
  2. Total anonymous requests has a limit per 24 hours.

As already deciphered by previous comments, one now needs to have an account and use the user and API keys with each API request to make it work.

I have seen that grabber is sending user_id as username instead of the numeric ID as per the API how to in gelbooru.

But for now it seems they have blocked API access for even authenticated usage.

brazenvoid commented 2 years ago

If only gelbooru offered a paid subscription and hassle free usage, I would take it up any day,

brazenvoid commented 2 years ago

Found this in gelbooru FAQ:

Q: My program that uses the Gelbooru API stopped working suddenly. What changed? A: Make sure that you are sending your login cookies when accessing the API. We no longer allow anonymous access to the API.

Bionus commented 2 years ago

Seems like their API is available for anonymous usage again, so I'll close this issue. However, in general I'd recommend logging into any source you're using to prevent this kind of issue from popping up.

As for the 100-image issue, see #2765.

I have seen that grabber is sending user_id as username instead of the numeric ID as per the API how to in gelbooru.

Not sure what you mean, Grabber sends user_id as whatever the user put in the settings in the "through URL" mode:

image