search

evilhero / mylar

An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
GNU General Public License v3.0
976 stars 172 forks source link

NZB Search Error #781

Closed Mortis7432 closed 9 years ago

Mortis7432 commented 10 years ago

Hello, I'm getting this error when trying to download through DOG NZB Api

Uncaught exception: Traceback (most recent call last): File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\logger.py", line 141, in new_run old_run(_args, _kwargs) File "C:\Python27\lib\threading.py", line 761, in run self.__target(_self.__args, _self.__kwargs) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 1407, in searchforissue foundNZB, prov = search_init(comic['ComicName'], result['Issue_Number'], str(ComicYear), comic['ComicYear'], Publisher, IssueDate, StoreDate, result['IssueID'], AlternateSearch, UseFuzzy, ComicVersion, SARC=None, IssueArcID=None, mode=mode, rsscheck=rsscheck, ComicID=result['ComicID'], filesafe=comic['ComicName_Filesafe']) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 213, in search_init findit = NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDate, StoreDate, searchprov, prov_count, IssDateFix, IssueID, UseFuzzy, newznab_host, ComicVersion=ComicVersion, SARC=SARC, IssueArcID=IssueArcID, ComicID=ComicID) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 633, in NZB_SEARCH stamp = time.mktime(issue_convert.timetuple()) OverflowError: mktime argument out of range

evilhero commented 10 years ago

Can you post the portion of the mylar.log file? It looks like it's being passed an improperly formatted date when it shouldn't be.

Mortis7432 commented 10 years ago

Do you want a screenshot of the log page or is there a way to export a file?

evilhero commented 10 years ago

You'd have to just copy and paste from the mylar.log file itself. It's all text, so it should be fairly easy.

The log screen just shows informational level logs, the log file holds all the debugging information.

Mortis7432 commented 10 years ago

Thanks for looking into this. 

 

I've put the log files here. 

 

https://www.dropbox.com/sh/5exv2jg0xa9vxb8/AAAwn45lV946f9SOqSQsVOkAa

 

Let me know if you can access them ok. 

 

Sent: Monday, August 04, 2014 at 10:55 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

You'd have to just copy and paste from the mylar.log file itself. It's all text, so it should be fairly easy.

The log screen just shows informational level logs, the log file holds all the debugging information.

— Reply to this email directly or view it on GitHub.

Mortis7432 commented 10 years ago

Slightly different error today.

Uncaught exception: Traceback (most recent call last): File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\Mylar.py", line 197, in main() File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\Mylar.py", line 175, in main mylar.start() File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylarinit.py", line 1293, in start rsscheck.tehMain() File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\rsscheck.py", line 52, in tehMain mylar.search.searchforissue(rsscheck='yes') File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 1407, in searchforissue foundNZB, prov = search_init(comic['ComicName'], result['Issue_Number'], str(ComicYear), comic['ComicYear'], Publisher, IssueDate, StoreDate, result['IssueID'], AlternateSearch, UseFuzzy, ComicVersion, SARC=None, IssueArcID=None, mode=mode, rsscheck=rsscheck, ComicID=result['ComicID'], filesafe=comic['ComicName_Filesafe']) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 194, in search_init findit = NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDate, StoreDate, searchprov, prov_count, IssDateFix, IssueID, UseFuzzy, newznab_host, ComicVersion=ComicVersion, SARC=SARC, IssueArcID=IssueArcID, RSS="yes", ComicID=ComicID) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 633, in NZB_SEARCH stamp = time.mktime(issue_convert.timetuple()) OverflowError: mktime argument out of range

evilhero commented 10 years ago

Looks like it's getting hung up on the result from nzbindex.nl.. I have a few hrs this afternoon (my Mylar time if you will), so I'll see if I can replicate what's happening. The date looks good coming down from nzbindex.nl, so it might be the date itself for the issue.

Can you check your Batman (1940) series and see what the issue date is for issue 74? That seems to be the initial issue it's having problems with.

Mortis7432 commented 10 years ago

Here is the info for Dark Knight #180

 

https://www.dropbox.com/s/56jzrjrplr63cw8/mylar.png

 

Sent: Tuesday, August 05, 2014 at 3:16 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

Looks like it's getting hung up on the result from nzbindex.nl.. I have a few hrs this afternoon (my Mylar time if you will), so I'll see if I can replicate what's happening. The date looks good coming down from nzbindex.nl, so it might be the date itself for the issue.

Cam you check you Batman: Legends of the Dark Knight series and see what the issue date is for issue 180?

— Reply to this email directly or view it on GitHub.

evilhero commented 10 years ago

Heh you're too quick.. I had just changed the offending comic.. It's issue 74 from Batman (1940). I'm betting the issue date shown is invalid, meaning it's 1952-12-00 if I had to guess (on comicvine it's listed as December 1952, not the full date which Mylar is expecting).

The good news, if you're on the latest development commit you can edit the issue dates now. Simply click on the offending date and enter in the new date (in the format of yyyy-mm-dd or else it won't accept it), so if you put in 1952-12-01 then hit enter it will accept it. You shouldn't get that error again for that particular issue anyways - I'll see about having Mylar auto fix issues that are showing as this.. If that's what the problem is (I'm just guessing on all of this)

Mortis7432 commented 10 years ago

No problems :) Keen to get this sorted got 3k wanted issues and counting. 

 

Batman (1940) issue 74 is showing in Mylar as 1952-12-01

 

Sent: Tuesday, August 05, 2014 at 3:35 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

Heh you're too quick.. I had just changed the offending comic.. It's issue 74 from Batman (1940). I'm betting the issue date shown is invalid, meaning it's 1952-12-00 if I had to guess (on comicvine it's listed as December 1952, not the full date which Mylar is expecting).

— Reply to this email directly or view it on GitHub.

evilhero commented 10 years ago

Ok so if the date in the issue details is correct, then it's a problem with the universal date conversion although it should be working based on those logs.

I'm just assuming you're on the latest development build, and that your system date on your os is set to English?

If yes to both, I'll have to look into this in a few hrs when I'm in front of my machine to see what's happening.

Mortis7432 commented 10 years ago

No problems. 

 

Mylar Version: development -- source build 7076adccfb06932eee584538b935b5c147125685.Python Version : 2.7.5

 

Server is set to uk.pool.ntp.org

 

Sent: Tuesday, August 05, 2014 at 3:51 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

Ok so if the date in the issue details is correct, then it's a problem with the universal date conversion although it should be working based on those logs.

I'm just assuming you're on the latest development build, and that your system date on your os is set to English?

If yes to both, I'll have to look into this in a few hrs when I'm in front of my machine to see what's happening.

— Reply to this email directly or view it on GitHub.

evilhero commented 10 years ago

Ok, this is weird - on my local repo:

05-Aug-2014 13:37:07 - DEBUG   :: Thread-13 : checking search result: Batman and the Outsiders 002 (1983) (digital) (Son of Ult$
05-Aug-2014 13:37:07 - DEBUG   :: Thread-13 : Thu, 12 Jun 2014 18:17:35 +0200 is after store date of 1952-10-08

Which is the search & actual result that yours is bombing out on (the 1952-10-08 is the actual store date, the publication date is what's listed on the GUI screen). So what's happening from what I can tell, Mylar can't parse the issue date properly and is using the store date (it does this on purpose), however your store date either doesn't exist or is wrong.

Can you refresh the Batman series ? What may have happened, is if this is a series you added a while ago and some commits have happened between then and now it may be missing some of the required fields to continue. Doing a Refresh Series usually is enough to bring things back in line - otherwise you'd have to delete / readd the series and see if that works.

Also, once you encounter an error (even if it continues trucking along), it's best to restart Mylar just to ensure that things are cleared out and everything can load in properly. Otherwise, strange things start to happen which normally can't be duplicated.

Mortis7432 commented 10 years ago

See output from Batman refresh. 

 

2014-08-05 19:05:23 INFO Update complete 2014-08-05 19:05:18 INFO I have added 0 new issues for this series that were not present before. 2014-08-05 19:05:18 INFO In the process of converting the data to CV, I changed the status of 715 issues. 2014-08-05 19:04:45 ERROR Uncaught exception: Traceback (most recent call last): File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\logger.py", line 141, in new_run old_run(_args, _kwargs) File "C:\Python27\lib\threading.py", line 761, in run self.__target(_self.__args, _self.kwargs) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 1407, in searchforissue foundNZB, prov = search_init(comic['ComicName'], result['Issue_Number'], str(ComicYear), comic['ComicYear'], Publisher, IssueDate, StoreDate, result['IssueID'], AlternateSearch, UseFuzzy, ComicVersion, SARC=None, IssueArcID=None, mode=mode, rsscheck=rsscheck, ComicID=result['ComicID'], filesafe=comic['ComicName_Filesafe']) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 213, in search_init findit = NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDate, StoreDate, searchprov, prov_count, IssDateFix, IssueID, UseFuzzy, newznab_host, ComicVersion=ComicVersion, SARC=SARC, IssueArcID=IssueArcID, ComicID=ComicID) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\search.py", line 521, in NZB_SEARCH time.sleep(pause_the_search) TypeError: a float is required 2014-08-05 19:04:44 INFO pausing for 111111111111111111111111111111111111111111111111111111111111 seconds before continuing to avoid hammering 2014-08-05 19:04:44 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using nzb.su 2014-08-05 19:04:44 INFO Could not find Issue 659 of Superman(1939) using dognzb 2014-08-05 19:04:37 INFO [ComicVine API] Comicvine API count now at : 10 / 400 in 4.78 minutes. 2014-08-05 19:04:18 ERROR Uncaught exception: Traceback (most recent call last): File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\logger.py", line 141, in new_run old_run(_args, *_kwargs) File "C:\Python27\lib\threading.py", line 761, in run self.target(_self.__args, *_self.__kwargs) File "C:\Users\Mortis\Desktop\mylar-development\mylar-development\mylar\webserve.py", line 912, in queueissue raise cherrypy.HTTPRedirect("comicDetails?ComicID=%s" % ComicID) HTTPRedirect: (['http://DL-PC:8090/comicDetails?ComicID=796'], 303) 2014-08-05 19:04:18 INFO Finished searching via :api. Issue not found - status kept as Wanted. 2014-08-05 19:04:18 INFO Could not find Issue 701 of Batman(1940) using OMGWTFNZBS 2014-08-05 19:04:18 WARNING Error fetching data from newznab: unknown url type: api.omgwtfnzbs.org/api?t=search&q=batman%20701%20&o=xml&cat=7030&apikey=bd35be8dfb7f828d9fd1674b149347ce&maxage=2048 2014-08-05 19:03:44 INFO pausing for 60 seconds before continuing to avoid hammering 2014-08-05 19:03:44 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using dognzb 2014-08-05 19:03:44 INFO Could not find Issue 659 of Superman(1939) using KAT 2014-08-05 19:03:43 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using KAT 2014-08-05 19:03:43 INFO Could not find Issue 659 of Superman(1939) using experimental 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using experimental 2014-08-05 19:03:42 INFO Finished searching via :rss. Issue not found - status kept as Wanted. 2014-08-05 19:03:42 INFO Could not find Issue 659 of Superman(1939) using OMGWTFNZBS 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using OMGWTFNZBS(newznab) [RSS] 2014-08-05 19:03:42 INFO Could not find Issue 659 of Superman(1939) using nzb.su 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using nzb.su [RSS] 2014-08-05 19:03:42 INFO Could not find Issue 659 of Superman(1939) using dognzb 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using dognzb [RSS] 2014-08-05 19:03:42 INFO Could not find Issue 659 of Superman(1939) using KAT 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using KAT [RSS] 2014-08-05 19:03:42 INFO Could not find Issue 659 of Superman(1939) using experimental 2014-08-05 19:03:42 INFO Shhh be very quiet...I'm looking for Superman issue: 659 (2007) using experimental [RSS] 2014-08-05 19:03:42 INFO search provider order is ['Experimental', 'kat', 'dognzb', 'nzb.su', 'newznab:OMGWTFNZBS'] 2014-08-05 19:03:41 INFO Initiating NZB Search scan at requested interval of 360 minutes. 2014-08-05 19:03:18 INFO pausing for 60 seconds before continuing to avoid hammering 2014-08-05 19:03:18 INFO Shhh be very quiet...I'm looking for Batman issue: 701 (2010) using OMGWTFNZBS (newznab) 2014-08-05 19:03:18 INFO Could not find Issue 701 of Batman(1940) using nzb.su 2014-08-05 19:03:10 INFO returning to dbupdate module 2014-08-05 19:03:10 INFO Updating complete for: Batman

 

Sent: Tuesday, August 05, 2014 at 6:54 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

Ok, this is weird - on my local repo:

05-Aug-2014 13:37:07 - DEBUG :: Thread-13 : checking search result: Batman and the Outsiders 002 (1983) (digital) (Son of Ult$ 05-Aug-2014 13:37:07 - DEBUG :: Thread-13 : Thu, 12 Jun 2014 18:17:35 +0200 is after store date of 1952-10-08

Which is the search & actual result that yours is bombing out on (the 1952-10-08 is the actual store date, the publication date is what's listed on the GUI screen). So what's happening from what I can tell, Mylar can't parse the issue date properly and is using the store date (it does this on purpose), however your store date either doesn't exist or is wrong.

Can you refresh the Batman series ? What may have happened, is if this is a series you added a while ago and some commits have happened between then and now it may be missing some of the required fields to continue. Doing a Refresh Series usually is enough to bring things back in line - otherwise you'd have to delete / readd the series and see if that works.

Also, once you encounter an error (even if it continues trucking along), it's best to restart Mylar just to ensure that things are cleared out and everything can load in properly. Otherwise, strange things start to happen which normally can't be duplicated.

— Reply to this email directly or view it on GitHub.

evilhero commented 10 years ago

Ok, so there are several errors in there:

The pause-between sleep error is possibly due to having either 0 or blank or some other non-numeric value in the Configuraiton / Web Interface / Search Delay field in the configuration GUI (or else you made changes to the configuration and didn't restart Mylar - an outstanding issue is that after any change made to the configuration and it's updated, Mylar needs to be restarted).

The 'Error fetching data from newznab: unknown url type: api.omgwtfnzbs.org/api?' is due to not have the url start with either the 'http://' or 'https://'

Because of those errors (especially the time-sleep error), you're not getting any search results processed and won't until it's fixed. I thought the time-sleep error was addressed a while ago, but I'm looking at this now since it's obviously happening again.

Thanks for the detailed logs!

EDIT. I'll push out something right now that should address the time-sleep problem. Looks like it was taking the string equivalent and multiplying it by 60, instead of just doing the math. This should fix the time-sleep error...

Mortis7432 commented 10 years ago

Ok, 

 

Updated with https:// will see how the API for OMGWTF works.

 

Updated with Search Delay, is there a minimum it should be for best results?

 

I had a slight amount of comics come in with the latest forced search (12x batman 2010 & 2011) but it seems to have stalled again for the other outstanding 4,000

 

Sent: Tuesday, August 05, 2014 at 7:17 PMFrom: evilhero notifications@github.comTo: evilhero/mylar mylar@noreply.github.comCc: Mortis7432 neildenham@gmx.comSubject: Re: [mylar] NZB Search Error (#781)

Ok, so there are several errors in there:

The pause-between sleep error is possibly due to having either 0 or blank or some other non-numeric value in the Configuraiton / Web Interface / Search Delay field in the configuration GUI (or else you made changes to the configuration and didn't restart Mylar - an outstanding issue is that after any change made to the configuration and it's updated, Mylar needs to be restarted).

The 'Error fetching data from newznab: unknown url type: api.omgwtfnzbs.org/api?' is due to not have the url start with either the 'http://' or 'https://'

Because of those errors (especially the time-sleep error), you're not getting any search results processed and won't until it's fixed. I thought the time-sleep error was addressed a while ago, but I'm looking at this now since it's obviously happening again.

Thanks for the detailed logs!

— Reply to this email directly or view it on GitHub.

evilhero commented 10 years ago

The minimum for the search delay is 1 minute - it's the absolute minimum I would recommend, anything lower (which you'd have to edit the code to do) would cause problems with your provider cause of the hammering - if you really want to be cautious and you're not in a rush, you could set it for 2 minutes even which is more than an adequate amount of cushion for any nzb provider I would imagine.

The stalling could be due to the time.sleep coming up again and your provider order. Only your newznab providers have the anti-hammering enforced, torrent searches and Experimental search have no such restriction thus far - however, I know from experience that if you are making constant requests to the Experimental option it will start to return 0 results as it has it's own anti-hammering method that it enforces and getting X requests / second from a specific IP would flag it.