evilhero / mylar

An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
GNU General Public License v3.0
976 stars 172 forks source link

Can no longer search for new series #1139

Closed skywayskase closed 9 years ago

skywayskase commented 9 years ago

Starting this morning, when I try to search for a new series (or add from the weekly pull-list), I get a "500 Internal server error" with the following readout:

"Traceback (most recent call last): File "C:\Program Files (x86)\Mylar\cherrypy_cprequest.py", line 645, in respond response.body = self.handler() File "C:\Program Files (x86)\Mylar\cherrypy\lib\encoding.py", line 188, in call self.body = self.oldhandler(_args, _kwargs) File "C:\Program Files (x86)\Mylar\cherrypy_cpdispatch.py", line 29, in call return self.callable(_self.args, _self.kwargs) File "C:\Program Files (x86)\Mylar\mylar\webserve.py", line 200, in searchit searchresults, explicit = mb.findComic(name, mode, issue=issue) File "C:\Program Files (x86)\Mylar\mylar\mb.py", line 132, in findComic searched = pullsearch(comicapi, comicquery, 0, explicit, type) File "C:\Program Files (x86)\Mylar\mylar\mb.py", line 71, in pullsearch file = urllib2.urlopen(PULLURL) File "C:\Python27\lib\urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "C:\Python27\lib\urllib2.py", line 437, in open response = meth(req, response) File "C:\Python27\lib\urllib2.py", line 550, in http_response 'http', request, response, code, msg, hdrs) File "C:\Python27\lib\urllib2.py", line 469, in error result = self._call_chain(_args) File "C:\Python27\lib\urllib2.py", line 409, in _call_chain result = func(_args) File "C:\Python27\lib\urllib2.py", line 656, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "C:\Python27\lib\urllib2.py", line 431, in open response = self._open(req, data) File "C:\Python27\lib\urllib2.py", line 449, in _open '_open', req) File "C:\Python27\lib\urllib2.py", line 409, in _call_chain result = func(*args) File "C:\Python27\lib\urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection, req) File "C:\Python27\lib\urllib2.py", line 1200, in do_open r = h.getresponse(buffering=True) File "C:\Python27\lib\httplib.py", line 1074, in getresponse response.begin() File "C:\Python27\lib\httplib.py", line 415, in begin version, status, reason = self._read_status() File "C:\Python27\lib\httplib.py", line 371, in _read_status line = self.fp.readline(_MAXLINE + 1) File "C:\Python27\lib\socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) error: [Errno 10054] An existing connection was forcibly closed by the remote host"

I checked the log file and found the following related error:

"02-Oct-2015 05:15:22 - DEBUG :: WEEKLYCHECK : Exception generated in thread WEEKLYCHECK: [Errno 10054] An existing connection was forcibly closed by the remote host 02-Oct-2015 05:15:22 - DEBUG :: WEEKLYCHECK : 'Traceback (most recent call last):\n File "C:\Program Files (x86)\Mylar\mylar\scheduler.py", line 78, in runAction\n self.action.run()\n File "C:\Program Files (x86)\Mylar\mylar\weeklypullit.py", line 30, in run\n mylar.weeklypull.pullit()\n File "C:\Program Files (x86)\Mylar\mylar\weeklypull.py", line 179, in pullit\n pullitcheck()\n File "C:\Program Files (x86)\Mylar\mylar\weeklypull.py", line 796, in pullitcheck\n statusupdate = updater.upcoming_update(ComicID=ComicID, ComicName=ComicName, IssueNumber=ComicIssue, IssueDate=ComicDate, forcecheck=forcecheck)\n File "C:\Program Files (x86)\Mylar\mylar\updater.py", line 373, in upcoming_update\n cchk = mylar.importer.updateissuedata(ComicID, ComicName, calledfrom=\'weeklycheck\')#mylar.importer.addComictoDB(ComicID,mismatch,pullupd)\n File "C:\Program Files (x86)\Mylar\mylar\importer.py", line 1143, in updateissuedata\n comic = cv.getComic(comicid, \'comic\')\n File "C:\Program Files (x86)\Mylar\mylar\cv.py", line 151, in getComic\n dom = pulldetails(comicid, \'comic\', None, 1)\n File "C:\Program Files (x86)\Mylar\mylar\cv.py", line 85, in pulldetails\n file = urllib2.urlopen(PULLURL)\n File "C:\Python27\lib\urllib2.py", line 154, in urlopen\n return opener.open(url, data, timeout)\n File "C:\Python27\lib\urllib2.py", line 437, in open\n response = meth(req, response)\n File "C:\Python27\lib\urllib2.py", line 550, in http_response\n \'http\', request, response, code, msg, hdrs)\n File "C:\Python27\lib\urllib2.py", line 469, in error\n result = self._call_chain(_args)\n File "C:\Python27\lib\urllib2.py", line 409, in _call_chain\n result = func(_args)\n File "C:\Python27\lib\urllib2.py", line 656, in http_error_302\n return self.parent.open(new, timeout=req.timeout)\n File "C:\Python27\lib\urllib2.py", line 431, in open\n response = self._open(req, data)\n File "C:\Python27\lib\urllib2.py", line 449, in _open\n \'_open\', req)\n File "C:\Python27\lib\urllib2.py", line 409, in _call_chain\n result = func(*args)\n File "C:\Python27\lib\urllib2.py", line 1227, in http_open\n return self.do_open(httplib.HTTPConnection, req)\n File "C:\Python27\lib\urllib2.py", line 1200, in do_open\n r = h.getresponse(buffering=True)\n File "C:\Python27\lib\httplib.py", line 1074, in getresponse\n response.begin()\n File "C:\Python27\lib\httplib.py", line 415, in begin\n version, status, reason = self._read_status()\n File "C:\Python27\lib\httplib.py", line 371, in _read_status\n line = self.fp.readline(_MAXLINE + 1)\n File "C:\Python27\lib\socket.py", line 476, in readline\n data = self._sock.recv(self._rbufsize)\nerror: [Errno 10054] An existing connection was forcibly closed by the remote host\n'"

Any idea what happened?

evilhero commented 9 years ago

Looks like you hit the 24/72hr ban thing that comicvine is enforcing as of late. There's no rhyme or reason for it happening, no warnings and actually very little that can be done atm by myself as they're enforcing these types of bans with no explanation so it's really hard to try and fix anything that might be causing it as no one knows why.

I've read posts that people think that mylar is the cause because it hammers the api or something - which it doesn't. It has a limiter that after 400 api hits in 15 minutes it stops api requests. Same with the db update option, it fires off 1 request every 10+ seconds which by any math is far below the 400 hits.

If I were to guess I think it's affecting users that use multiple apps that utilize the same api key. Since there's no way to track usage across different apps (mylar only tracks what it uses and limits based on that usage) I think the hits are going over said limit repeatidly and they're getting banned. So if you're using mylar and the usage hits say 200-300 hits in 15 mins, and then you're using a tagger to do other stuff it's very easy to go over the limit they try to enforce.

Will check into this more though and keep an eye on their forums for any additional information but atm I don't think there's anything that can be done as this type of ban/limit breaks anything that has to do with the api usage.

skywayskase commented 9 years ago

Hmmm, interesting. I do use the same api for comic tagger, but I haven't been manually tagging anything in the last couple of days. Does your limiter take mylars usage of comic tagger when automatically post-processing into consideration?

evilhero commented 9 years ago

Well no not perse, it does a general counter increment (meaning the command gets issued and the count goes up 1 but there's no indication if it does more than one api call within the comictagger program itself). The cv api has no way to poll what your current usage is so it falls on the app to try and keep track the best it can.

evilhero commented 9 years ago

Apparently has been fixed (CV data center issue was blocking all api requests as malicious).

skywayskase commented 9 years ago

Yep, I just tested it again this morning and everything is working a-okay again.