search

RipMeApp / ripme

Downloads albums in bulk
MIT License
3.72k stars 631 forks source link

[NOT SOLVED!] Derpibooru.org doesnt work - Remember System #1851

Closed socalledkion closed 3 years ago

socalledkion commented 3 years ago

*Informations: Derpibooru does not seem to work. It would be great if you could add that. It would also be nice to have a remember cookie system behind it so that you can bypass the filter.

Thanks in advance ~ Kion

rautamiekka commented 3 years ago

I need to warn that Derpibooru provides a very functional non-slow API which doesn't hide much anything (you need an acc and the API key from your settings my bad, you don't need the API key to bypass 18+ restrictions if the filter ID is specified) and AFAIK doesn't have any call limits, which'd be a better idea than scanning the website.

^ Sure the website doesn't change next to at all, but I'm sure they're providing the API to lessen the load on their systems.

If all else fails, gallery-dl is your best bet. The config is very involved, though, it's JSON and the config docs format requires using some brain cells.

socalledkion commented 3 years ago

If all else fails, gallery-dl is your best bet. The config is very involved, though, it's JSON and the config docs format requires using some brain cells.

First of all, thank you for the reply. Unfortunately I am a complete 0 at something like this and would like to integrate this into RIPME, is there a possibility?

rautamiekka commented 3 years ago

If all else fails, gallery-dl is your best bet. The config is very involved, though, it's JSON and the config docs format requires using some brain cells.

First of all, thank you for the reply. Unfortunately I am a complete 0 at something like this and would like to integrate this into RIPME, is there a possibility?

Not without a whole new OAuth 2.0 infrastructure, pretty sure. I just checked to refresh my memory, and noticed Derpibooru had been supported before, but it was a mix of HTML and JSON parsing, both direct HTTP requests; dunno if the direct JSON requests are possible anymore.

With the minute or 2 long checking through the code I couldn't find if RipMe is able to visit another page to click a button and return to another to scan the page, so there's that.

socalledkion commented 3 years ago

If all else fails, gallery-dl is your best bet. The config is very involved, though, it's JSON and the config docs format requires using some brain cells.

First of all, thank you for the reply. Unfortunately I am a complete 0 at something like this and would like to integrate this into RIPME, is there a possibility?

Not without a whole new OAuth 2.0 infrastructure, pretty sure. I just checked to refresh my memory, and noticed Derpibooru had been supported before, but it was a mix of HTML and JSON parsing, both direct HTTP requests; dunno if the direct JSON requests are possible anymore.

With the minute or 2 long checking through the code I couldn't find if RipMe is able to visit another page to click a button and return to another to scan the page, so there's that.

Does it mean it won't work? Or will not be possible?

rautamiekka commented 3 years ago

^ With API, not possible ATM IIUC. With website scanning, after fixing the ripper.

socalledkion commented 3 years ago

^ With API, not possible ATM IIUC. With website scanning, after fixing the ripper.

I would have been so happy if it had worked out. Thanks for your answers.

a-washing-machine commented 3 years ago

I might be wrong, but honestly, comparing the JSON outputs for the old and new version, it doesn't look like that much of a difference exists for the parts relevant for downloading the images (see the "representations" tag).

Heck, just changing the query, and slightly altering getImageUrlFromJson should be enough to fix it???

Pattern p = Pattern.compile("^https?://derpibooru\\.org/search\\.json\\?q=([^&]+).*?$");

    private String getImageUrlFromJson(JSONObject json) {
        return "https:" + json.getJSONObject("representations").getString("full");
    }

While it is indeed a great tool, in this case I'd really rather not use mikf's gallery-dl for this and start rebuilding my entire collection all over.

OLD JSON: { "id": 1482281, "created_at": "2017-07-09T14:32:50.791Z", "updated_at": "2019-11-09T14:51:43.935Z", "first_seen_at": "2017-07-09T14:32:50.791Z", "tags": "absurd file size, animated, artist:khuzang, black background, cake, chocolate, chocolate cake, do you want this cake, equestria girls, family guy, female, food, harry partridge, nicolas cage, safe, seizure warning, simple background, sound, sunset satan, sunset shimmer, twilight sparkle, webm, youtube link", "tag_ids": [ 988, 21013, 22118, 22973, 22975, 25318, 26446, 26921, 27141, 27821, 29701, 35984, 40482, 41133, 41769, 42470, 46192, 48122, 64220, 80107, 89484, 98505, 122193 ], "uploader_id": 389301, "score": 104, "comment_count": 5, "width": 1280, "height": 720, "tag_count": 23, "file_name": "Sunset_Shimmers_Wants_Cake.webm", "description": "", "uploader": "northern haste", "image": "//derpicdn.net/img/view/2017/7/9/1482281__safe_artist-colon-khuzang_sunset+shimmer_twilight+sparkle_equestria+girls_absurd+file+size_animated_black+background_cake_chocolate_chocolate+.webm", "upvotes": 107, "downvotes": 3, "faves": 76, "aspect_ratio": 1.7777777777777777, "original_format": "webm", "mime_type": "video/webm", "sha512_hash": "somelonghashstringhere", "orig_sha512_hash": "somelonghashstringhere", "source_url": "https://www.youtube.com/watch?v=68CDQdxiai8", "representations": { "thumb_tiny": "//derpicdn.net/img/2017/7/9/1482281/thumb_tiny.webm", "thumb_small": "//derpicdn.net/img/2017/7/9/1482281/thumb_small.webm", "thumb": "//derpicdn.net/img/2017/7/9/1482281/thumb.webm", "small": "//derpicdn.net/img/2017/7/9/1482281/small.webm", "medium": "//derpicdn.net/img/2017/7/9/1482281/medium.webm", "large": "//derpicdn.net/img/2017/7/9/1482281/large.webm", "tall": "//derpicdn.net/img/2017/7/9/1482281/tall.webm", "full": "//derpicdn.net/img/view/2017/7/9/1482281.webm", "webm": "//derpicdn.net/img/2017/7/9/1482281/full.webm", "mp4": "//derpicdn.net/img/2017/7/9/1482281/full.mp4" }, "is_rendered": true, "is_optimized": true, "interactions": [], "spoilered": true }

NEW JSON: { "image": { "faves": 76, "description": "", "score": 104, "comment_count": 5, "wilson_score": 0.8985886280237884, "tag_ids": [ 988, 21013, 22118, 22973, 22975, 25318, 26446, 26921, 27141, 27821, 29701, 35984, 40482, 41133, 41769, 42470, 46192, 48122, 64220, 80107, 89484, 98505, 122193 ], "duplicate_of": null, "upvotes": 107, "downvotes": 3, "orig_sha512_hash": "somelonghashstringhere", "uploader_id": 389301, "first_seen_at": "2017-07-09T14:32:50", "aspect_ratio": 1.7777777777777777, "processed": true, "name": "Sunset_Shimmers_Wants_Cake.webm", "representations": { "full": "https://derpicdn.net/img/view/2017/7/9/1482281.webm", "large": "https://derpicdn.net/img/2017/7/9/1482281/large.webm", "medium": "https://derpicdn.net/img/2017/7/9/1482281/medium.webm", "small": "https://derpicdn.net/img/2017/7/9/1482281/small.webm", "tall": "https://derpicdn.net/img/2017/7/9/1482281/tall.webm", "thumb": "https://derpicdn.net/img/2017/7/9/1482281/thumb.webm", "thumb_small": "https://derpicdn.net/img/2017/7/9/1482281/thumb_small.webm", "thumb_tiny": "https://derpicdn.net/img/2017/7/9/1482281/thumb_tiny.webm" }, "created_at": "2017-07-09T14:32:50", "id": 1482281, "tags": [ "animated", "black background", "cake", "chocolate", "chocolate cake", "do you want this cake", "equestria girls", "family guy", "female", "food", "harry partridge", "nicolas cage", "safe", "seizure warning", "simple background", "sound", "twilight sparkle", "youtube link", "sunset shimmer", "sunset satan", "artist:khuzang", "absurd file size", "webm" ], "uploader": "northern haste", "source_url": "https://www.youtube.com/watch?v=68CDQdxiai8", "hidden_from_users": false, "width": 1280, "height": 720, "thumbnails_generated": true, "format": "webm", "spoilered": true, "deletion_reason": null, "tag_count": 23, "intensities": { "ne": 0.3148654603587963, "nw": 1.427989626736111, "se": 2.5334808032407407, "sw": 3.151060383969907 }, "view_url": "https://derpicdn.net/img/view/2017/7/9/1482281__safe_artist-colon-khuzang_sunset+shimmer_twilight+sparkle_equestria+girls_absurd+file+size_animated_black+background_cake_chocolate_chocolate+.webm", "mime_type": "video/webm", "sha512_hash": "somelonghashstringhere", "updated_at": "2019-11-09T14:51:43" } }