search

muldjord / skyscraper

Powerful and versatile game scraper written in c++
GNU General Public License v3.0
489 stars 128 forks source link

[HELP] Scrape didn't finish, can't use cache ? #295

Closed zzedmore closed 3 years ago

zzedmore commented 3 years ago

@muldjord Hi, your scraper is awesome but I have a problem. I was doing a scrape from ArcadeDB grabbing covers/flyers. I got 1986 out of 2013 covers (which took nearly 3 hours on my slow connection!) but Putty disconnected. If I do Skyscraper -p arcade -s arcadedb again then it starts downloading and overwriting from the beginning again ! Now i'm left with 1986 files in the cache with non-standard names, is there any way I can resume this or make it use the almost complete cache ?

muldjord commented 3 years ago

For future reference, please direct your questions here. This is for bug reports only.

But this once, I will answer: You will need to start scraping from the beginning. If Skyscraper is cut off without allowing it to close down properly, the xml database with the cached resources isn't written to disc. You will have the files, but they are useless since there is no database entries telling Skyscraper what they are.

If you have an unstable connection I suggest always running Skyscraper through screen. That way you can just reconnect to it if Putty disconnects. Do a Google search for "Linux screen command" or something like that. Or if you already have it installed, you can read the manual for it with man screen. Use q to quit the manual.

Good luck!