search

cbanack / comic-vine-scraper

An add-on script for ComicRack that lets you copy details from Comic Vine into your comic books.
255 stars 48 forks source link

Use rate limiter on API calls #423

Closed duckpuppy closed 8 years ago

duckpuppy commented 8 years ago

This does not force a delay on every call - it actually uses a decorator to only wait long enough to ensure a 1.5 second delay between calls.

Clearmist commented 8 years ago

duckpuppy, I swapped in your cvconnection.py and tested the plugin. I like that it is slightly faster than waiting 1.5 seconds for every call. I was able to scrape about 20 books before hitting the exceeded rate limit error.

fieldhouse commented 8 years ago

I'm getting a consistent crash on a title. Haven't looked at it too much yet to figure out why... ======> scraping next comic book: 'Aliens Vampirella 003(2015)(Digital Exclusive Editionl)(TLK-EMPIRE-HD).cbr' trying to match this book automatically... ------------------- PYTHON ERROR ------------------------ Caught AttributeError: 'module' object has no attribute '_query_issue_details_page' Traceback (most recent call last): File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 143, in scrape File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 257, in _ScrapeEngine__scrape File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 426, in _ScrapeEngine__scrape_book File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\automatcher.py", line 55, in find_series_ref File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\db.py", line 222, in query_issue File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 464, in _query_issue

and ======> scraping next comic book: 'All for Love v1 05 (1948 Prize) (c2c) (Soothsayr-Yoc-Novus).cbz' trying to match this book automatically... ------------------- PYTHON ERROR ------------------------ Caught AttributeError: 'module' object has no attribute '_query_issue_details_page' Traceback (most recent call last): File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 143, in scrape File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 257, in _ScrapeEngine__scrape File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 426, in _ScrapeEngine__scrape_book File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\automatcher.py", line 55, in find_series_ref File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\db.py", line 222, in query_issue File "...\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 464, in _query_issue

duckpuppy commented 8 years ago

Yeah, I don't think the rate limiter on the ComicVine side is very accurate. It certainly takes way more than one second between API calls to reliably not trigger the rate limiting.

cbanack commented 8 years ago

Thanks for your help with this, I've integrated changes based on your suggestions into release 1.0.91.

Clearmist commented 8 years ago

You are welcome, Cory. I am glad you and the ComicVine administrators were able to get the bugs worked out of their system. I'm glad you put more work in to a project you feel under appreciated for. It looks like Duck (the other guy who issued a pull request) showed interest in moving the official release to his fork and taking over. I hope that works out. I like this project and I am glad you created it!

-Matthew On Nov 28, 2015 12:56, "Cory Banack" notifications@github.com wrote:

Thanks for your help with this, I've integrated changes based on your suggestions into release 1.0.91.

— Reply to this email directly or view it on GitHub https://github.com/cbanack/comic-vine-scraper/pull/423#issuecomment-160332197 .