Error accessing Comic Vine

ghost commented 2 years ago

Unable to scrape using Comic Vine scrapper for the last 12-24 hours. Last worked on Nov 1 around 7:30pm CT. Using Version 1.0.100 Getting error message starting at 7:30 am Nov 2nd.

"The Comic Vine online database could not be reached. This may be due to temporary technical error with the database, or there may be a problem with your internet connection."

This seems to be the same issue as last time where Comic Vine made a change to their API code. I have rebooted my system and tried again several times before posting this and have tried updating multiple files. I have access to the internet and Comic Vine.

I posted on Reddit also to see if others are having the same issue.

nosoop4u commented 2 years ago

Getting the same. Status page at https://comicvine.gamespot.com/api/ shows "You have no current rate limits...," so it's not that I in particular am being usage throttled. Also no posts about new API features/changes at https://comicvine.gamespot.com/forums/api-developers-2334/.

I don't believe there's any logs or debug settings in the scraper, so if you're not a coder that can troubleshoot the actual code, and ComicVine truly changed something in the API that borked the scraper, we're out of luck until the developer or someone else can troubleshoot the underlying code.

SenorSmartyPants commented 2 years ago

trying to match this book automatically...
ERROR OCCURRED CONTACTING COMICVINE. RETRYING...
------------------- PYTHON ERROR ------------------------
Caught DatabaseConnectionError: Comic Vine database could not be reached
url: http://comicvine.gamespot.com/api/search/?api_key=...&client=cvscraper&format=xml&limit=100&resources=volume&field_list=name,start_year,publisher,id,image,count_of_issues&query=firefly%20-%20river%20run
CAUSE: System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.Scripting.Interpreter.FuncCallInstruction`2.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.LightLambda.Run3[T0,T1,T2,TRet](T0 arg0, T1 arg1, T2 arg2)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at IronPython.Compiler.Ast.CallExpression.Invoke0Instruction.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0 arg0, T1 arg1)
   at Microsoft.Scripting.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
   at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
Traceback (most recent call last):
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 142, in scrape
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 257, in _ScrapeEngine__scrape
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 426, in _ScrapeEngine__scrape_book
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\automatcher.py", line 38, in find_series_ref
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\automatcher.py", line 75, in __find_best_series
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\db.py", line 156, in query_series_refs
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 142, in _query_series_refs
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 165, in __query_series_refs
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 65, in _query_series_ids_dom
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 217, in __get_dom
  File "C:\Users\FakeUser\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 174, in __get_dom

nosoop4u commented 2 years ago

Hey, so I pasted that query into my browser, replaced the "..." with my API key and I got a valid XML response, and not a 403 forbidden error. But still getting the error when running any query thru the scraper. Not quite sure what's going on.

Where did you see that log message, SenorSmartyPants?

url: http://comicvine.gamespot.com/api/search/?api_key=...&client=cvscraper&format=xml&limit=100&resources=volume&field_list=name,start_year,publisher,id,image,count_of_issues&query=firefly%20-%20river%20run

SenorSmartyPants commented 2 years ago

You have to enable the console with a command line option. I forget the code at the moment.

WillPresley commented 2 years ago

I wonder if CV is banning or rate-limiting the scraper user-agent? Is there still a way to set that manually in Advanced Settings? It has been a long time since I've messed with any of this, but I'm having the same issue today. Just a repeat of the last major problem in #471 really.

christophershad commented 2 years ago

I am getting the same error, I am showing as not rate-limited in the API page browsing manually, Get the error even if scraping a single comic, also no API changes noted in https://comicvine.gamespot.com/forums/api-developers-2334/

cbanack commented 2 years ago

You can also press, Ctrl+Shift+L to get a copy of the current log (scraper window must have keyboard focus when you do this.)

christophershad commented 2 years ago

Tried the rate-limit fix of SCRAPE_DELAY=8 (and other numbers, up to 20) in advanced, no dice, same error 403 forbidden in the logs.

christophershad commented 2 years ago

Cross-posted this to the ComicVine API developer forum

gmickd commented 2 years ago

I found a fix to this on a Reddit post.

if you uncomment line 378 and 379 of utils.py it should work again.

# request.UserAgent = "[ComicVineScraper, version " + \
#   Resources.SCRIPT_VERSION + "]"

credit to /u/Bloodwah.

cbanack commented 2 years ago

Seems like Comic Vine has changed their minds (again) about which User-Agent they will accept.

I'll do my best to get a new version 1.0.101 out with the fix that you guys have so helpfully found. Sometime soon, unless life interferes -- until then, the fix that gmickd reported will work, or you can just downgrade to version 1.0.99 for now.

cbanack commented 2 years ago

This issue should be fixed in ComicVineScraper v1.0.101, which you can get here (or from the regular download page.)

Cheers!