search

danbooru / danbooru

A taggable image board written in Rails.
Other
2.26k stars 416 forks source link

IQDB is broken #3758

Closed BrokenEagle closed 6 years ago

BrokenEagle commented 6 years ago

The IQDB page isn't working. https://danbooru.donmai.us/iqdb_queries

When it gets to IQDBS, it gives an HTTP 401 error.

This page isn’t working
If the problem continues, contact the site owner.
HTTP ERROR 401

The Find Similar links on post pages also aren't working. https://danbooru.donmai.us/iqdb_queries?post_id=3165488

A few other notes.

  1. IQDB has been removed from the site map and the uploads listing

So there's no way to navigate there unless you know the link.

  1. It seems that the data is being sent form-encoded and not as part of the URL

This will make it impossible to use SauceNAO's image finder tool with Danbooru. http://saucenao.com/tools/

r888888888 commented 6 years ago

I forgot I had to deploy IQDBS.

The link is still present on the site map, it's just been renamed to Similar Image Search (since IQDB wasn't a very descriptive name before).

I didn't realize SauceNAO had a tool. I'll try to add support back in.

BrokenEagle commented 6 years ago

Yeah, on the Firefox version you can use form data (I believe), but on the Chrome version, you only can specify the URL. The following is what the configuration for Danbooru used to be on the Chrome version.

image

So it basically takes the URL that you specify, and adds on the URL encoded image path after that.

Also, I just tried a search, and this time I did get a response from IQDBS, but then it just left me on that page (https://karasuma.donmai.us/similar) with just the IQDBS JSON data.

[{"post_id":2569454,"width":128,"height":128,"score":95.808468},{"post_id":3165488,"width":128,"height":128,"score":94.721983}]

It also doesn't seem to working when using the Bookmarklet, as it gets stuck with "Loading similar...".

nonamethanks commented 6 years ago

@r888888888 it's still broken btw. Now I get 500 Internal Server Error on all queries.

Direct url search returns a different error like here.

r888888888 commented 6 years ago

Some fixes in 56ee44dc0091b8342c64e7627149effcfde93005

The URL builder is fixed. A redirect for /iqdb_queries/check was added.

BrokenEagle commented 6 years ago

Yeah, the standalone page seems to be working now. I checked both the post_id and url addons, and they both seem to be working. I also checked the form data for the URL and file option and they seem to be working as well.

So I believe the only things that need fixing are:

1. Find similar on uploads page

Response: 404

<html>
<head>
    <title>Page not found</title>
    <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"/>
    <link rel="stylesheet" type="text/css" media="screen" href="/assets/application.css" title="default" />
</head>

<body>
    <div style="text-align: center;">
        <h1>That page does not exist</h1>
        <p><a href="/">Return to index</a></p>
    </div>
</body>
</html>

2. Find similar link on post page

Response: 404 Same response as 1 above.

BrokenEagle commented 6 years ago

As another issue that needs to be fixed, it seems like the IQDB queries page does not handle either the XML or the JSON formats. This one is kind of critical for me, as I use Danbooru's IQDB API a lot with my image scanner. Luckily I have a backup for using IQDB.org, but it would be preferential to use Danbooru's since it doesn't have the rate limiting that IQDB.org does.

nonamethanks commented 6 years ago

Same actually. My scraper to find uploadable material relied on the old query to check for revisions and whether the file on danbooru was of better quality (mostly by checking for height/width and source link).

Is the old format going to still be available, or will it be necessary to make an extra api request for each post returned as match now?

r888888888 commented 6 years ago

I will add a JSON response.

The key check on IQDBS was removed and that should still render JSON (assuming you don't pass in a callback) so you should be able to just query against https://karasuma.donmai.us/similar directly. It only returns post id+score so you would need to cross reference against Danbooru for file size and dimensions.

BrokenEagle commented 6 years ago

DeusExCalamus said (forum #148286): The similar image search on the upload page is still broken, any ETA?

The find similar on the post page is also still broken, but that is also probably not as critical as the upload one, which can affect whether a user uploads an image or not.

BrokenEagle commented 6 years ago

Looks like similar image search on the upload page is working. Also, the Similar button on the post edit page works.

However, the Find similar link on the sidebar under Options still isn't working though. I find that functionality useful myself, as it can be used as a simple way to search for potential parent/child relations.

BrokenEagle commented 6 years ago

Found another issue with IQDB that deals with using the API through Javascript on other sites. Because of the redirect to IQDBS (karasuma.donmai.us), it's running into some CORS violations.

Failed to load https://karasuma.donmai.us/similar?callback=https%3A%2F%2Fdanbooru.donmai.us%2Fiqdb_queries.json&url=https%3A%2F%2Fcs.sankakucomplex.com%2Fdata%2Fpreview%2F76%2F90%2F7690d8c5464bbe5f926ded3d75a39458.jpg
Redirect from 'https://karasuma.donmai.us/similar?callback=https%3A%2F%2Fdanbooru.donmai.us%2Fiqdb_queries.json&url=https%3A%2F%2F%2Fdata%2Fpreview%2F76%2F90%2F7690d8c5464bbe5f926ded3d75a39458.jpg' to 'https://danbooru.donmai.us/iqdb_queries.json?matches=%5B%5D' has been blocked by CORS policy
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access.

The above comes from a script I developed for use on other sites such as the other Boorus and Chans, which helps me and other users of the script quickly and easily weed out images that have already been uploaded.

Before the change to IQDB, it worked just fine thanks to CORS being enabled sitewide in issue #3350. Perhaps enabling it on IQDBS (karasuma.donmai.us) would fix it as well, or are multi-redirects that Danbooru currently uses for IQDB not allowed at all off-site?

BrokenEagle commented 6 years ago

Some of the recent refactors are causing issues with the uploads page.

image

So it's showing the IQDB queries page instead of a shorter version to only show the results. It may also be causing issues with uploading by file when the bookmarklet is also used, as one user stated that it was asking them to log in. Perhaps the two file inputs are causing conflicts with each other when both are rendered on the page?

BrokenEagle commented 6 years ago

Actually, I got a clarification on the above from the user.

fossilnix said: I'm using the bookmarklet: I open the image in a tab, then click the bookmarklet. It loads, including the weird similar images search. The image URL shows in both the "Source" box and the Similar "URL" box. I type in tags in the "Tags" box and press "Enter" (or click "Submit" blow the Tags, it does the same thing) and instead of uploading like usual, it prompts me to log in

So perhaps the two submit buttons are causing conflicts with each other?

r888888888 commented 6 years ago

I believe most of the outstanding issues related to IQDB are resolved now.

The callback is only used for direct submissions from the HTML page so it shouldn't affect bookmarklets that pass along the URL or post id directly.

BrokenEagle commented 6 years ago

Yeah, the only possible issue is that the Find similar link on the post page now navigates to a new page instead of opening a popup dialogue on the same page. However that doesn't bother me so much, though I'll keep an ear out for it to see if anyone does have an issue with it.