evilhero / mylar

An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
GNU General Public License v3.0
977 stars 173 forks source link

nzbX.co support #126

Closed digiwombat closed 11 years ago

digiwombat commented 11 years ago

They've been added in to CouchPotato and Headphones mainlines

https://nzbx.co/docs

Just thought it might be worth taking a look at as far as search providers. Not sure how big of a pain in the ass adding providers is, but people seem to be liking this one. 100k users already.

evilhero commented 11 years ago

I'm not against the idea of new providers (if they're not newznab-based), but from the looks of it I don't see any comics being indexed at nzbx.co ?

digiwombat commented 11 years ago

They are still doing their backlog run (supposed to take another 4 days) so there are a few that are missing, but their category system isn't quite right yet. Searching will give you comic results and there is a category listing for it internally:

https://nzbx.co/s?q=Uncanny+X-men&i=releases&l=50&sf=1&sb=age&st=desc#

You can see they're scraping a.b.comics.dcp at least:

https://nzbx.co/d?e5f3fd6dc6b2601d5472a8b4f7f90e72

In the API calls their group IDs are 123 (a.b.comics.dcp), 126 (a.b.pictures.comics.dcp), 127 (a.b.pictures.comics.repost).

Though, hmmm.... HMMMM. Their API doesn't have a clear way to sort by groupid until after you get the JSON call. Sort of a pain in the ass there. Hm.

EDIT: Forgot these two: 128 alt.binaries.pictures.comics.reposts 129 alt.binaries.pictures.comics.complete

Edit number twwwwwooo: Looks like it does have a cat endpoint. Cat is standard, 7030. The GET variable Q has to be set or it will return an empty JSON

https://nzbx.co/forum/topic/id/9

ALRIGHT! Final edit:

My suggestion here may have been a bit premature since I've had trouble squeezing anything useful out of their comics section for a while. While it seems like getting things implemented shouldn't be too hard (and it's a public index), I am not sure what order their pulls are running, but it certainly seems like comics are low down the list. Going to bother them on the forums and see what the story is.

LemonadeDev commented 11 years ago

So, what groups do you want us to index in order of priority? Let me know and I'll get it taken care of.

digiwombat commented 11 years ago

Generally in order of importance it goes:

a.b.comics.dcp a.b.pictures.comics.dcp (these two are new releases, updated the most often, generally the most important) a.b.pictures.comics.complete (less updated, full comic sets) a.b.comics.reposts a.b.pictures.comics.reposts (these two are reposts, good for things that have fallen out of retention)

Thanks for the response! :D

LemonadeDev commented 11 years ago

These should be updating properly now (as of right now), I'll work on a backfill for these groups tomorrow and will update you.

digiwombat commented 11 years ago

Looks like more results are starting to fill in. It's still missing some a lot of things other NewzNab providers have (Infinite Vacation, X-Men Schism, and a couple of others) but things have definitely filled in over the past day or so and looks to be going in the right direction, so I am officially submitting my plea for them to be added to Mylar now. :thumbsup:

evilhero commented 11 years ago

Nzbx is included now as a provider but haven't had much of a chance to test it because if the problems they had with searches (I stole headphones logic for this one because it was json based)...I tried one really quick today and it pulled it down and post processed so hopefully its working.

digiwombat commented 11 years ago

Not sure if it's part of the ongoing Windows-is-shit saga, but nzbX is giving some errors:

http://i.imgur.com/D3JJ5.png

Not sure if it's malformed on the way in on or the way out but certainly something is rotten in the state of search.py

evilhero commented 11 years ago

Sorry that one's my fault...search.py didn't have the code to deal with a "no results" from nzbx. I fixed it and will havr the commit up shortly hopefully.

digiwombat commented 11 years ago

Still getting some less than delicious results from NZBx.co! This time it errored when it found a comic to send over and gave multiple no results hits (which could be fine, not sure)

http://i.imgur.com/CZrL0.png

evilhero commented 11 years ago

That ones not an error with nzbx...I had changed the priority in sab in the config layout. Go into your Config and in the Sabnzbd section set a priority level for downloads (default is typical). Save the config, restart Mylar and you should be good.

I'll look into this as this will affect everyone :(

digiwombat commented 11 years ago

Alright, that fixed that up, but now I'm hitting PostProcessing errors on NZBx downloads. Again! Not sure if this is NZBx related, but do want to keep things tidy if it is.

http://i.imgur.com/9uLYx.png

Log output: 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Found X-Treme X-Men 007 (2013) (c2c) (Meganubis-Novus). Size: 36.0 MB 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Found X-Treme X-Men 007 (2013) (Meganubis-Novus). Size: 27.0 MB 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Found X-Treme X-Men 007 (2013) (Digital) (Nahga-Empire). Size: 17.0 MB 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : nzbx.co! 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : checking search result: X-Treme X-Men 007.1 (2013) (Meganubis-Novus) 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Entry: X-Treme X-Men 007.1 (2013) (Meganubis-Novus) 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Cleantitle: x-treme x-men 007 1 (2013) (meganubis-novus) 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 0. Bracket Word: x-treme x-men 007 1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Comic: x-treme x-men 007 1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 1. Bracket Word: 2013 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : year detected: 2013 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 2012 - not right - years do not match 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : further analysis reveals this was published inbetween Nov-Jan, incrementing year to 2013 has resulted in a match! 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 3. Bracket Word: meganubis-novus 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : original nzb comic and issue: x-treme x-men 007 1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : adjusted nzb comic and issue: x treme x men 007 1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : chg_comic:x treme x men 007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : changeup to decimal: .1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : adjusting from: x treme x men 007 1 to: 007.1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ['x', 'treme', 'x', 'men', '007', '1'] nzb series word count: 4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ['X', 'Treme', 'X', 'Men'] watchlist word count: 4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : length match..proceeding 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : search-length: 4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Watchlist-length: 4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : splitit: x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 0 Comparing: X .to. x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : word matched on : x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : splitit: treme 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 1 Comparing: Treme .to. treme 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : word matched on : treme 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : splitit: x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 2 Comparing: X .to. x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : word matched on : x 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : splitit: men 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : 3 Comparing: Men .to. men 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : word matched on : men 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : scount:4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : splitit-len:4 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : we got 100.0 percent. 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : it's a go captain... - we matched 100.0%! 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : this should be a match! 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : issue we are looking for is : 7.1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : integer value of issue we are looking for : 7010 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Found IssueNumber: 007.1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ..before decimal: 007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...after decimal: 1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : issue we found for is : 007.1 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : integer value of issue we are found : 7010 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : issues match! 13-Jan-2013 10:48:58 - INFO :: Thread-13 : Found X-Treme X-Men (2012) issue: 7.1 using nzbx 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : link given by: nzbx 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : link: https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : linkforapi: https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : cache directory successfully found at : C:\Users\randall\Desktop\SickBeard\omniverse\cache 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : link to retrieve via api:https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : nzbname used for post-processing:X-Treme.X-Men.007.1.(2013).(Meganubis-Novus) 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : send-to-SAB host string: http://localhost:8080 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...selecting API type: http://localhost:8080/api?mode=addurl&name= 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...attaching nzb provider link: http://localhost:8080/api?mode=addurl&name=https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...setting priority: http://localhost:8080/api?mode=addurl&name=https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007&priority=0 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...attaching category: http://localhost:8080/api?mode=addurl&name=https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007&priority=0&cat=comics 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : ...attaching rename script: http://localhost:8080/api?mode=addurl&name=https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007&priority=0&cat=comics&script=ComicRN.py 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Completed send-to-SAB link: http://localhost:8080/api?mode=addurl&name=https://nzbx.co/nzb?9c8e4d7c24be17ae07f9a3520e8de9e7*|*X-Treme%20X-Men%20007&priority=0&cat=comics&script=ComicRN.py&apikey=ff162285bfd3fe892bf7e3690cc91323 13-Jan-2013 10:48:58 - INFO :: Thread-13 : Successfully sent nzb file to SABnzbd 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : we should break out now - sucessful search previous 13-Jan-2013 10:48:58 - DEBUG :: Thread-13 : Found matching comic...preparing to send to Updater with IssueID: 372458 and nzbname: X-Treme.X-Men.007.1.(2013).(Meganubis-Novus) 13-Jan-2013 10:48:59 - DEBUG :: Thread-13 : findit = found! 13-Jan-2013 10:48:59 - DEBUG :: Thread-13 : updating status to snatched 13-Jan-2013 10:48:59 - DEBUG :: Thread-13 : updating snatched db. 13-Jan-2013 10:48:59 - INFO :: Thread-13 : Updating now complete for X-Treme X-Men issue: 7.10

That's everything from the find, and the postprocessing dump from SABnzbd:

Loading config from C:\Program Files (x86)\SABnzbd\user-scripts\autoProcessComics.cfg Opening URL: http://localhost:8090/post_process?nzb_name=X-TremeX-Men007&nzb_folder=C%3A%5CUsers%5Crandall%5CDocuments%5CDownloads%5Ccomplete%5CX-TremeX-Men007

It's the same and has failed for everything. Saga, X-Treme X-men, and a few others. I'm going to fly out on a limb and assume it's the lack of spaces in the folder names? Maybe? Since that is the biggest change I have noticed.

I do NOT have Replace Spaces checked in my config file. Just in case that matters.

evilhero commented 11 years ago

Actually it looks like nzbx.co is renaming the file as it gets sent to SAB via directlink ... "X-Treme X-Men 007.1 (2013)(Meganubis-Novus)" becomes "XtremeX-Men007" for some reason.

Not quite sure what's goin on - I'll do some testing on my end to see, but I think this is the same issue that was happening with nzb.su previously.

Ok I figured it out...everything after the *| is the designated filename...should be an easy fix (i hope)...next commit :)

evilhero commented 11 years ago

The latest commit has it working (take your poison - master / development)...I tested on several and it actually worked without a problem (that's me though)

digiwombat commented 11 years ago

Up on the latest dev, working like smooth paste, so I will make this one closed since I can't seem to find any faults with search/download/post on nzbx now.

Thanks a ton for adding this in and thanks to LemonadeDev for getting the groups filled out! :D Really helped fill in some blank spots that experimental wasn't on top of and my Newznab people were behind on/stingy with the API calls.

GOOD WORK! :cake: