Newznab plugin does not parse categories properly from SpotWeb Newznab API

[forkless]

ERRORLOG:

Error during _gatherCategories of Newznab(Default) Error: No JSON object could be decoded

Traceback (most recent call last): File "/share/MD0_DATA/.qpkg/XDM/source/xdm/web/ajax.py", line 70, in pluginCall status, data, msg = p_function(_fn_args) File "/share/MD0_DATA/.qpkg/XDM/source/plugins/Indexer/Newznab-de_lad1337_newznab-0_6/Newznab.py", line 142, in _gatherCategories for cat in r.json()['categories']['category']: File "/share/MD0_DATA/.qpkg/XDM/source/lib/requests/models.py", line 629, in json return json.loads(self.text or self.content, *_kwargs) File "/opt/lib/python2.7/json/init.py", line 326, in loads return _default_decoder.decode(s) File "/opt/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

NEWZNAB XML FORMATTING:

<caps>
<server version="0.1" title="Spotweb" strapline="Spotweb API Index" email="test@test.com (test admin)" url="http://127.0.0.1/spotweb/" image="http://127.0.0.1/spotweb/images/spotnet.gif"/>
<limits max="500" default="100"/>
<registration available="no" open="no"/>
<searching>
<search available="yes"/>
<tv-search available="yes"/>
<movie-search available="yes"/>
<audio-search available="yes"/>
</searching>
<categories>
<category id="1000" name="Console">
<subcat id="1010" name="NDS"/>
<subcat id="1020" name="PSP"/>
<subcat id="1030" name="Wii"/>
<subcat id="1040" name="Xbox"/>
<subcat id="1050" name="Xbox 360"/>
<subcat id="1080" name="PS3"/>
</category>
<category id="2000" name="Movies">
<subcat id="2030" name="SD"/>
<subcat id="2040" name="HD"/>
<subcat id="2060" name="Sport"/>
</category>
<category id="3000" name="Audio">
<subcat id="3010" name="MP3"/>
<subcat id="3020" name="Video"/>
<subcat id="3040" name="Lossless"/>
</category>
<category id="4000" name="PC">
<subcat id="4030" name="Mac"/>
<subcat id="4040" name="Phone"/>
<subcat id="4050" name="Games"/>
</category>
<category id="5000" name="TV">
<subcat id="5020" name="Foreign"/>
<subcat id="5030" name="SD"/>
<subcat id="5040" name="HD"/>
<subcat id="5050" name="Other"/>
<subcat id="5060" name="Sport"/>
</category>
<category id="6000" name="XXX">
<subcat id="6010" name="DVD"/>
<subcat id="6020" name="WMV"/>
<subcat id="6030" name="XviD"/>
<subcat id="6040" name="x264"/>
</category>
<category id="7000" name="Other">
<subcat id="7020" name="Ebook"/>
</category>
</categories>
</caps>

[lad1337]

the newznab plugin only supports the output as json ... it seams the api syntax is different for SpotWeb or maybe you need to do this https://github.com/spotweb/spotweb/wiki/Spotweb-als-Newznab-Provider ? i cant find a documentation for the api

[forkless]

I've done that already. I use spotweb as an alternative newznab provider for CouchPotato, Sickbeard et al. which works fine.

Eg. http://127.0.0.1/spotweb/api?t=c works. One dfifference I do notice with the same api call to my newznab provider on http://127.0.0.1/newznab/api?t=c it will give slightly different XML (it adds groups & genre taxonomies).

[lad1337]

the issue is that the newznab plugin only supports json output newznab can do that apperently SpotWeb can't so either a newznab plugin for xml has to be made that will work with SpotWeb or the existing one has to take xml output into consideration ... but honestly xml is a pain

[forkless]

Spotweb supports JSON output it just defaults to XML as far as I know, it should listen to the o=JSON URI parameter

Maybe I shouldn't have mentioned XML in the issue comment/title, the only thing I intended to point out is that the newznab API is working when you call it with api?t=c (Which indeed produces XML, by the way it does this by default on Newznab itself as well).

Like I mentioned before CP and SB parse the content from the API. Could it be there is a bug in your JSON parser/the way you handle your stream?

EDIT: Never mind, I think I've found a bug in the Spotweb API

[forkless]

https://myspotweb.com/api?t=search&q=Linux&apikey=##MYAPIKEY##&o=json

[{"ID":"QqZtqmI59xsTOtuUgeNUL@spot.net","name":"Kerio Connect 8.2.0 Linux 64","size":"519432948","adddate":"2013-10-28 23:55:10","guid":"QqZtqmI59xsTOtuUgeNUL@spot.net","fromname":"whizkidz","completion":100,"comments":"0","category_name":"Applications: TUX","category_ids":[],"_totalrows":57},{"ID":"6DXlkA5PQCwJetuUg05lu@spot.net","name":"Kerio Connect 8.2.0 Linux 32","size":"535704523","adddate":"2013-10-28 23:54:32","guid":"6DXlkA5PQCwJetuUg05lu@spot.net","fromname":"whizkidz","completion":100,"comments":"0","category_name":"Applications: TUX","category_ids":[]},{"ID":"V5hHguOQUhQZpttUgkL3A@spot.net","name":"Kerio Connect 8.1.3 Linux 64","size":"261317540","adddate":"2013-10-28 00:02:10","guid":"V5hHguOQUhQZpttUgkL3A@spot.net","fromname":"whizkidz","completion":100,"comments":"0","category_name":"Applications: TUX","category_ids":[]},{"ID":"vuNqzWmt57M4JptUgJdcA@spot.net","name":"Kerio Connect linux 32","size":"267924996","adddate":"2013-10-27 23:59:55","guid":"vuNqzWmt57M4JptUgJdcA@spot.net","fromname":"whizkidz","completion":100,"comments":"0","category_name":"Applications: TUX","category_ids":[]},

Here's some sample JSON output from Spotweb, hope this will help

[domaximus]

any progress on this issue? I'm experiencing exactly the same.

[avdstelt]

I am trying to get it to work in a new Spotweb plugin with XML parsing.

[avdstelt]

I have parsing the nzb and category with XML working it only needs an extra lib. xmltodict i will put everything together and will do a pull request later this evening.

[lad1337]

@avdstelt did you upgrade the default Nezwnab plugin or make a new one ? because if its a new one you could host it yourself

[avdstelt]

It's a new one maybe you can help me how to host it i am pretty new to git :)

[lad1337]

i suspect https://github.com/avdstelt/XDM-main-plugin-repo is what you are using as a start there are two ways either you delte every other plugin except your own (locally) then commit the changes and push that way only your plugin / version is available there. when you did this you might want to change the name of the repo like "my xdm plugins"

or you could create a new repo on github (the make that part quite easy) and move the modified plugin to the local copy of the new repo

now to make your git(hub) repo into a XDM repo you have to modify the meta.json (the name is irrelevant) how you do that is up to you

• either write it by hand
• use the code icon on the plugin manage page on the loaded plugins to get the information on that plugin
• use the repoJSONmaker.py to create a json that fits all plugins in the supplied directory

please give the repoJSONmaker.py a chance and i can help you with the rest besides if you have any question feel free to ask either here @ https://xdm.userecho.com/ or on irc freenode.net on #xdm

[avdstelt]

Ok i will try it this evening i am also having some problems with the pluginRootLibarys is there a good documentation how to use this?

[lad1337]

this is all i have written down https://xdm.readthedocs.org/en/latest/plugin/index.html#pluginrootlibarys

[avdstelt]

I saw that, but at some point it doesn't work for me i created a directory pluginRootLibarys/xmltodict and put the file xmltodict.py in that directory.

When i write import xmltodict.xmltodict i am getting module xmltodict not found.

[lad1337]

The xmltodict folder does it have an inti.py if not add one Otherwise the folder is not a module It might also work to just add the xmltodict.py to the pluginRootLibarys folder (without the extra folder)

[avdstelt]

You are right there isn't one i will add one and see what happens then.

[avdstelt]

Having some problems with the download.type and downloader.types

In the Spotweb plugin type and types are both de.lad1337.nzb but when i print them in the tasks.py they are not the same?

I did changed them to nl.avdstelt.nzb in the new plugin when i started to build it.

[avdstelt]

Fixed it by deleting the entries in the database.

[avdstelt]

Ok check https://github.com/avdstelt/XDM-plugins for Spotweb plugin.

[avdstelt]

You can add my repos with the following url https://raw.github.com/avdstelt/XDM-plugins/master/meta.json

[avdstelt]

I think that this issue can be closed.