search

cbanack / comic-vine-scraper

An add-on script for ComicRack that lets you copy details from Comic Vine into your comic books.
244 stars 47 forks source link

Scraper not fetching extra covers #431

Closed SkaryMonk closed 4 months ago

SkaryMonk commented 8 years ago

After updating to 1.0.92 (ComicRack version 0.9.177), the scraper no longer finds any more covers, when I click "(more covers)". I'm simply told there's only a single cover.

A specific example would be Grimm Fairy Tales presents Red Riding Hood One-Shot (http://comicvine.gamespot.com/grimm-fairy-tales-presents-red-riding-hood-one-sho/4000-492312/). Zenescope never published anything in their life which only had one cover, which is what made me suspicious, and ComicVine obviously has more than one cover.

Let me know if you need any further information!

boshuda commented 8 years ago

This is an unfortunate necessity because of Comicvine's decision not to put the covers as fetchable via the API (something about deals with comic publishers). The way CVS used to do it was by scraping the web page. While this had technically been against the rules, Comicvine hadn't been checking for it until they changed their servers. They started blocking the IPs of people caught scraping their web pages. Search their API forum and you'll see a whole drama that unfolded within the last few months. Which is to say this will not be fixed until ComicVine changes their API to pass all of the covers along when requested.

theotocopulitos commented 8 years ago

A work around to this at some point might be to add in the found books list the direct link to the webpage for each book, so that clicking it would launch the internet browser to the comic page where the covers are displayed...

On Fri, Feb 26, 2016 at 1:02 PM, boshuda notifications@github.com wrote:

This is an unfortunate necessity because of Comicvine's decision not to put the covers as fetchable via the API (something about deals with comic publishers). The way CVS used to do it was by scraping the web page. While this had technically been against the rules, Comicvine hadn't been checking for it until they changed their servers. They started blocking the IPs of people caught scraping their web pages. Search their API forum and you'll see a whole drama that unfolded within the last few months.

— Reply to this email directly or view it on GitHub https://github.com/cbanack/comic-vine-scraper/issues/431#issuecomment-189246118 .

cbanack commented 8 years ago

I'm hoping that they actually add the alternate covers to their official API. They kinda suggested that they'd do it, but it hasn't happened yet.

Once they do, it'll be really easy to get this feature working again.

giotte commented 4 months ago

I know this is incredibly late in coming, but I thought I'd share that ComicVine does now, in fact, provide additional covers via the API. Not sure when it was added, but it definitely is there and works.

All you need to do is update the _query_issue function in cvdb.py so that it looks in the DOM for the "associated_images" element and then append any image URLs to the issue.image_urls_sl list. You can see an implementation of that here.

This is such a useful feature and I'm just glad that it's working again.

cbanack commented 4 months ago

Thanks for noticing this, giotte! I implemented your changes into a new version of ComicVineScraper (1.0.102) and it seems to be working great. It'll be worth it for people to update to this new version, since it makes "find more covers" feature work properly again, and it will also improve the scraper's ability to "automatically" choose the correct comic book series when scraping (for people who have that feature turned on.)

unclerandy1969 commented 2 months ago

I don’t know why, but for some reason I cannot install the latest (final) update. The only thing it seems to write in the scripts folder is a few image files

cbanack commented 2 months ago

Try removing the plugin, restarting comicrack fully (or restart your entire computer), then reinstall the plugin. And then restart comicrack again if needed

I seem to remember that there is a comicrack bug that interfered with installation of plugins sometimes. They was how people worked around it.

unclerandy1969 commented 2 months ago

Try removing the plugin, restarting comicrack fully (or restart your entire computer), then reinstall the plugin. And then restart comicrack again if needed

I seem to remember that there is a comicrack bug that interfered with installation of plugins sometimes. They was how people worked around it.

WOW! Thanks, that did the trick