Open mornhold opened 8 years ago
Apple240Bloom
commented
8 years ago it might be that you're trying to do too much rainbow dash + pinkie pie pulls down 30 thousand images.... I just got (rainbow dash, pinkie pie, image macro, animated) to work for 83 images using the executable joeyh3 put together, it's working fine.
mornhold
commented
8 years ago Hmm. Here is what the program. When i try to download the tag "twilight+sparkle" or "twilight sparkle" write error, and tag "fluttershy" downloaded normally.
2016-07-07 22:06:19,303 - DEBUG - Logging started. derpibooru_dl.py:572: UserWarning: gzip transfer encoding is experimental! 2016-07-07 22:06:21,513 - DEBUG - API Key looks fine. 2016-07-07 22:06:21,513 - INFO - Attempting to resume any failed downloads.
Thanks for the answer.
oblivion2k
commented
8 years ago I was just about to open a new issue for this when I saw somebody already has. I can confirm that the 2016-7-1 release isn't grabbing any results where tags have spaces in them.
'''2016-07-06 21:40:10,828 - DEBUG - Starting search for query: 'explicit,Princess+Luna' 2016-07-06 21:40:10,829 - INFO - Starting new HTTPS connection (1): derpiboo.ru 2016-07-06 21:40:11,207 - DEBUG - "GET /search.json?perpage=50&sf=created_at&q=explicit%2CPrincess%2BLuna&key=[SCRUBBED]&page=1&sd=desc HTTP/1.1" 200 None 2016-07-06 21:40:11,210 - WARNING - No submissions to save! Query:'explicit,Princess+Luna' 2016-07-06 21:40:11,211 - DEBUG - Erasing resume data pickle'''
I'll take a look at it in the morning if nobody gets to it first.
woodenphone
commented
8 years ago Maybe it's because spaces need to be escaped from " " to "%20"? To be honest this whole thing feels a bit overcomplicated and in need of a rewrite from the bottom up.
gsass1
commented
8 years ago Well, shouldn't the + signs be converted to a space?
oblivion2k
commented
8 years ago Spaces are tricky in most coding languages. Woodenphone probably has the right idea. I'll look into it soon.
oblivion2k
commented
8 years ago Looking into it now, I think I see the problem, shouldn't take long.
oblivion2k
commented
8 years ago Sorry guys, I got distracted by friends and Pokemon GO, lol. I thought I had fixed it, looks the changes they made to the API makes it so that spaces don't need to be escaped at all anymore; typing the search string that the derpibooru_dl would generate without space replacing returns valid results. Problem I'm running into now is that with that particular part of the code removed and query_list.append(replaced_line) changed to query_list.append(stripped_line), it's going into an HTTP200 None loop, while still showing a valid result in get_last_not_html.txt. However, the get_last_not_html.txt only shows the results of the testing search, searching just for the word 'explicit'. It's not actually completing the search query list part of the routine.
Here's the debug log recorded if I just search for 'princess luna' with my modified code:
'''2016-07-08 20:11:35,017 - WARNING - No submissions to save! Query:'from_list' 2016-07-08 20:11:35,019 - INFO - Now downloading user set tags/queries 2016-07-08 20:11:35,019 - INFO - Now proccessing query 1 of 25: 'Princess Luna' 2016-07-08 20:11:35,019 - DEBUG - Starting search for query: 'Princess Luna' 2016-07-08 20:11:35,032 - INFO - Starting new HTTPS connection (1): derpiboo.ru 2016-07-08 20:11:35,706 - DEBUG - "GET /search.json?perpage=50&sf=created_at&q=Princess+Luna&key=[EXPUNGED]&page=1&sd=desc HTTP/1.1" 200 None 2016-07-08 20:11:35,877 - INFO - Starting new HTTPS connection (1): derpiboo.ru 2016-07-08 20:11:36,371 - DEBUG - "GET /search.json?perpage=50&sf=created_at&q=Princess+Luna&key=[EXPUNGED]&page=2&sd=desc HTTP/1.1" 200 None 2016-07-08 20:11:36,566 - INFO - Starting new HTTPS connection (1): derpiboo.ru 2016-07-08 20:11:37,096 - DEBUG - "GET /search.json?perpage=50&sf=created_at&q=Princess+Luna&key=[EXPUNGED]&page=3&sd=desc HTTP/1.1" 200 None etc...'''
I've confirmed that putting that URL into a web browser (example: derpiboo.ru//search.json?perpage=50&sf=created_at&q=Princess+Luna&key=[EXPUNGED]&page=3&sd=desc HTTP/1.1) returns valid data, so I'm not sure at this point why the code isn't handling that data properly. And I just realized that it's incrementing the page number each time, so it's not actually a true loop. It's recognizing that valid data is there, and moving onto the next page, presumably until no pages are left, but it's not downloading any of the images. I haven't looked through the section of code that does that yet so maybe there's some code there that needs modified still.
I've ran out of time today, and tomorrow I'll be busy until late night/early morning, so Woodenphone if you have any ideas why my relatively simple change might have messed things up so badly, you might be able to fix it quicker than I will. If anybody else would like to tinker with it as well, it's very easy to get to the point I'm at now, just delete the 'def replaced_line' and associated subroutines, and make the replacement query_list.append that I mentioned above, and you'll get the same results I am. Good luck.
oblivion2k
commented
8 years ago Never mind haha, it works perfectly. It just doesn't display the same output during page indexing that it did in the 2014 version of the downloader, and I didn't know that, so I assumed something was wrong. Patched derpibooru_dl.py has been uploaded in my fork, and pull request sent to the master branch. You can view my summary of changes here, download the patched file here, or wait for Woodenphone to merge my patch into master. Have fun and remember to close the issue once you've confirmed it's working on your end mornhold.
mornhold
commented
8 years ago Thank you all for your help, but Version 2016-6-30 I have works fine, but the folder does not derpibooru_dl.py file, so I can not replace anything, as advised by oblivion2k.
Version 2016-7-1b when trying to download either writes
10/07/2016 14: 28: 41,811 - ERROR - 'Image' object has no attribute 'id_number'
Traceback (most recent call last):
File "derpibooru_dl.py", line 1415, in
Version 2016-7-1 has no .exe, and downloading for python, i was never able to figure out how to run it.
I decided to use the version 6.30.2016 that works for me, and to use the search manually.
Once again, I want to apologize for my stupidity and thank everyone for their help.
gsass1
commented
8 years ago Seems like the admins from Derpibooru even do not know what is going on. https://derpibooru.org/meta/api-update
mkody
commented
8 years ago @mornhold Something's wrong, you're not running 2016-7-1b, lol. Every id_number that did make the script crash were renamed to id on this release (that's a recent change in the Derpibooru API).
EDIT: This is not your fault, the release archive is probably the wrong one.
@Nuke928 Oh welp, bugs happens. At least Derpibooru APIs are not as bad as MyLittleFaceWhen's (there's no way to know how to use their API v3).
Sorry for the clumsy text, since I use an interpreter. Could you help me. I could not get to work version 2016-7-1 and 2016-7-1b. But 2016-06-30 version works fine. The only downside is it is not working properly download in derpibooru_dl_tag_list.txt, so it can not look like a combination of pinkie pie and rainbow dash, that is, search tags, consisting of 2 or more words. Please help fix this problem and thank you.