deepghs / waifuc

Efficient Train Data Collector for Anime Waifu
https://deepghs.github.io/waifuc/
MIT License
272 stars 23 forks source link

Program Terminates Abnormally Without Any Error Message #50

Closed Eugeoter closed 7 months ago

Eugeoter commented 7 months ago

The program usually terminates abnormally without ANY error message (looks hidden?), even the try-except syntax cannot capture it. This occurs when downloading, especially when starting another download task during crawling (unstable network). Every time this happens, I have to restart crawling and request those downloaded sources again...

Here is the error message.

C:\Users\15070\AppData\Local\Programs\Python\Python38\lib\site-packages\waifuc\source\web.py:51: UserWarning: Danbooru resource 7225057 unidentified as image, skipped.7, 17519.37s/it, tag=k-on!] 
  warnings.warn(f'{self.group_name.capitalize()} resource {id_} unidentified as image, skipped.')
DanbooruSource: 51it [01:43,  1.62s/it]
PS C:\Users\15070\Desktop\projects\waifuset>

I'd be appreciated if you could help me~

narugo1992 commented 7 months ago

waifuc do NOT have any exit like this.

so the only explanation is that your OS killed this process.

just check what is going on your OS.

narugo1992 commented 7 months ago

@Eugeoter wait, another possibility.

you said you start another download task, can you describe what is that exactly?

the another possibility is, if this "another download task" is danbooru, and too fast, the danbooru pagination api may down, by return empty data, and the waifuc process will stop (certainly without error or exception, because the empty pagination is quite normal when all the images ran out).

can u find the DanbooruSource's code, and add one print to what the pagination api return?

NOTE: do NOT try to make danbooru unhappy in any forms if you dont want your ip get banned forever 🤣

Eugeoter commented 7 months ago

@Eugeoter wait, another possibility.

you said you start another download task, can you describe what is that exactly?

the another possibility is, if this "another download task" is danbooru, and too fast, the danbooru pagination api may down, by return empty data, and the waifuc process will stop (certainly without error or exception, because the empty pagination is quite normal when all the images ran out).

can u find the 's code, and add one print to what the pagination api return?DanbooruSource

NOTE: do NOT try to make danbooru unhappy in any forms if you dont want your ip get banned forever 🤣

Thank you for your early reply! Such 'another download task' is just a usual task like downloading a game (not from danbooru). So, I think your formal answer sounds more possible though I can't figure out why my OS kill the program. Anyway, thank you for your amazing project and hoping it becomes better and better~

narugo1992 commented 7 months ago

@Eugeoter you are welcome. if you have some further finding or clues, feel free to reopen this issue. 😄