search

McSib / e621_downloader

E621 and E926 downloader made in the Rust programming langauge.
Apache License 2.0
63 stars 14 forks source link

Suggestion for better method of download #99

Open mithrilarmor opened 1 year ago

mithrilarmor commented 1 year ago

My tag list has probably grown unusually large (about 800 tags of artists and characters) and lately downloading randomly fails whether after grabbing just a few tags or after downloading begins anytime before it finishes. It could be my ISP or maybe e621.net got some form of protection from such requests. Could the way downloading works be changed to grabbing then downloading tag by tag instead of grabbing them all at once then downloading them all? I have 16GB of RAM just in case it might be important. By the way awesome work and functionality of this program, past year was great for archiving my favorite artists work.

Here are last few lines from the log file: 22:00:48 [ERROR] Unable to deserialize json to "alloc::vec::Vec"! 22:00:48 [TRACE] (1) e621_downloader::e621::sender: [src\e621\sender\mod.rs:393] Failed to perform bulk search...

McSib commented 1 year ago

It seems like the problem is not related to the program's download or memory usage. It appears to be a difficulty with correctly obtaining a specific tag. Would you mind sharing your tag file and log file with me? Don't worry, the log file only contains essential information for the program's functionality and does not include any personal details.

McSib commented 1 year ago

By the way awesome work and functionality of this program, past year was great for archiving my favorite artists work.

And no problem, I like working on this project in my spare time. Happy to see others enjoying the software 😄

mithrilarmor commented 1 year ago

That log file is long gone, the next day since posting the problem I have tried running it again without change and it worked, maybe it was a problem on their server. That problem occurred suddenly after starting the program and answering with "n" and within a minute an error message from first post shows and immediately the program exits.

Concerning invalid tags like artist that gets on DNP list, they could be set to ignore to prevent stopping the program and informing at the end of downloads which tags are invalid if any.

McSib commented 1 year ago

Concerning invalid tags like artist that gets on DNP list, they could be set to ignore to prevent stopping the program and informing at the end of downloads which tags are invalid if any.

I could definitely look into adding something like that, I need to figure out how to detect DNP artists or check for invalid tags more generally. If I can implement a better parsing system, I can see about implementing that kind of feature then.