The plugin doesn't install

[Pireo]

QBT 3.3.11, Windows 7 x64 ultimate, python 3.6.1

The plugin refuses to install and it says "not supported", with either weblink or local file.

[ArionMiles]

I'm on Python2.7 and it installs and works just fine. Please try downloading python2.7 and installing it.

[Pireo]

Ok, listen, sorry for the OT, but is it possible with these api to create a https://1337x.to/ plugin? https://github.com/c0b41/xtorrent https://github.com/PatrickE94/leetxapi

[ArionMiles]

1337x doesn't seem to have an RSS feed or any other form of official API. The APIs you've linked are written in JS. So unless you know how to incorporate JS api into your python search plugin, you can't use them. You can learn to use HTML.Parser module in python and use it to create the plugin. For more info, look at qBittorrent's Search Plugin writing guidlines.

[ArionMiles]

Was your issue solved by using Python2.7? If yes, I might have to check the cross-compatibility of the plugin with python 3.

[v1k45]

Sorry for jumping in middle. @Pireo, I just created a search plugin for 1337x.to, you can use it if you want :)

[Pireo]

@v1k45 thanks a lot, I've opened a little issue about it @ArionMiles I have both python 2 and 3 installed.

[ArionMiles]

I can't seem to find what's wrong. Using Python3, I cannot install it, but with python2, it works fine. For now, I've updated the Unofficial Search Plugins list stating it supports Python2.

[Pireo]

I've tried to change the "import" section with something else found on other plugins, and I was able to install it, but didn't show any search results. Anyway, thanks for creating this engine, the site has verified torrents and does search in file names too.

[v1k45]

Several module including urllib2 and StringIO were renamed in Python 3. I think that is the main cause or this issue.

[ArionMiles]

I added python3 module names and fallback for python2 modules and got the plugin to install. Searches are aborted. I think simultaneous python2 and python3 installations are causing an issue.

Running the plugin with nova2.py returns an error saying ImportError: no module named Parse on line 36 of nova2.py

[Pireo]

@ArionMiles I've tried zooqle with python 2 and while it works, it's limited to 30 results it seems. @v1k45 can you correct this? https://pastebin.com/AS7nC98F It has the same problem 1337x had, and only displays 10 results.

[v1k45]

@ArionMiles check qbittorrent's Python3 implementation of retrieve_url function, maybe you'll find what's going wrong.

@Pireo you code doesn't update list_searches attribute anywhere, the list remains empty , so the for loop is not executed in the end.

[ArionMiles]

@v1k45 I did inspect the retrieve_url function of nova3/nova2.py, but I can't test it because two python installations fuck up the testing. I think I'll close this issue and not add support for python3 at all.

[Pireo]

@ArionMiles but what about the 30 results issue?

[ArionMiles]

I'm unable to reproduce that issue. If anybody else faces it, I'll have another look.

[Pireo]

@v1k45 it wasn't done by me, I hope someone can fix it, so, am I the only one who gets only 30 results with zooqle?

[ArionMiles]

@Pireo open another issue for "Search results limited to 30 items." We'll resolve this over there. I'm acknowledging this issue, and working on a fix.

[dwilbanks]

I've got a version working that works directly under 3.4 python, I the important bits are:

try:
    from urllib2 import urlopen, Request, URLError
except ImportError:
    from urllib.request import urlopen, Request, URLError

For clarity, I've renamed your retrieve_url to retrieve_url_gzip

[dwilbanks]

oops, one more thing that's important:

from io import StringIO

I found a page that describes compatibility http://python-future.org/compatible_idioms.html

[ArionMiles]

I've already added python3 module names and python2 fallback. This enables the plugin to be installed in qbittorrent, but the script doesn't return search results. If your script returns results, can you post your script?

[dwilbanks]

yes, it's fully functional now, might have missed something I'm sorry I don't know how to use github to give you my changes

#VERSION: 1.10
#AUTHOR: Arion_Miles (https://github.com/ArionMiles/)
#LICENSE: MIT License
from xml.dom import minidom
from novaprinter import prettyPrinter
from helpers import download_file 
from io import StringIO
import gzip

try:
    from urllib2 import urlopen, Request, URLError
except ImportError:
    from urllib.request import urlopen, Request, URLError

def retrieve_url_gzip(url):
    """ Return the content of the url page as a string """
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0'}
    req = Request(url, headers = headers)
    try:
        response = urlopen(req)
    except URLError as errno:
        print(" ".join(("Connection error:", str(errno.reason))))
        print(" ".join(("URL:", url)))
        return ""
    dat = response.read()
    # Check if it is gzipped
    if dat[:2] == '\037\213':
        # Data is gzip encoded, decode it
        compressedstream = StringIO.StringIO(dat)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        extracted_data = gzipper.read()
        dat = extracted_data
        return dat
    return dat #new line

class zooqle(object):
    """ Search engine class """
    url = 'https://zooqle.com'
    name = 'Zooqle'
    supported_categories = {'all'       : 'all',
                            'movies'    : 'Movies',
                            'tv'        : 'TV',
                            'music'     : 'Music',
                            'games'     : 'Games',
                            'anime'     : 'Anime',
                            'software'  : 'Apps',
                            'books'     : 'Books',
                            'others'  : 'Other'}

    def download_torrent(self, info):
        """ Downloader """
        print(download_file(info))

    def search(self, what, cat="all"):
        """ Performs search """
        query = "".join((self.url, "/search?q=", what, "+category%3A", self.supported_categories[cat], "&fmt=rss"))
        response = retrieve_url_gzip(query)
        xmldoc = minidom.parseString(response)
#        itemlist = xmldoc.getElementsByTagName('opensearch:totalResults')

        itemlist = xmldoc.getElementsByTagName('item')
        for item in itemlist:
            zooqle_dict = zooqle_dict = {"engine_url" : self.url}
            zooqle_dict['name'] = item.getElementsByTagName('title')[0].childNodes[0].data
            zooqle_dict["link"] = item.getElementsByTagName('enclosure')[0].attributes['url'].value
            zooqle_dict["desc_link"] = item.getElementsByTagName('link')[0].childNodes[0].data
            zooqle_dict["size"] = item.getElementsByTagName('enclosure')[0].attributes['length'].childNodes[0].data
            zooqle_dict["leech"] = item.getElementsByTagName('torrent:peers')[0].childNodes[0].data
            if not zooqle_dict["leech"].isdigit():
                zooqle_dict["leech"] = ''
            zooqle_dict["seeds"] = item.getElementsByTagName('torrent:seeds')[0].childNodes[0].data
            if not zooqle_dict["seeds"].isdigit():
                zooqle_dict["seeds"] = ''
            prettyPrinter(zooqle_dict)
        return

[dwilbanks]

        totalResultElt = xmldoc.getElementsByTagName('opensearch:totalResults')[0]
        totalResultVal = totalResultElt.childNodes[0].nodeValue

[ArionMiles]

you can edit your comments. where does the latest comment's code fit in the main code? I still can't test the script because of two python installations.

[dwilbanks]

it does not fit anywhere, it's the solution to more than 30 results, it's still a work in progress. I thought by seeing the code you'd know what it meant.

Are you saying that my code does not work in 2.x? I thought it would work in both.

[ArionMiles]

You've changed the urllib and StringIO names in the script for python3, but the retrieve_url_gzip function still has python2 names for these libraries. This is the retrieve_url function for python3 (found in nova3/helpers.py):

def retrieve_url(url):
    """ Return the content of the url page as a string """
    req = urllib.request.Request(url, headers = headers)
    try:
        response = urllib.request.urlopen(req)
    except urllib.error.URLError as errno:
        print(" ".join(("Connection error:", str(errno.reason))))
        return ""
    dat = response.read()
    # Check if it is gzipped
    if dat[:2] == b'\x1f\x8b':
        # Data is gzip encoded, decode it
        compressedstream = io.BytesIO(dat)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        extracted_data = gzipper.read()
        dat = extracted_data
    info = response.info()
    charset = 'utf-8'
    try:
        ignore, charset = info['Content-Type'].split('charset=')
    except:
        pass
    dat = dat.decode(charset, 'replace')
    dat = htmlentitydecode(dat)
    #return dat.encode('utf-8', 'replace')
    return dat

[dwilbanks]

in 2.x the Request object is located inside urllib2 its full name is urllib2.Request in 3.x the Request object is located inside urllib.request its full name is urllib.request.Request

When you use the format below: from [parent] import [child]

you no longer refer to your objects using their full names, you can use their shortened names.

I took a guess at the location of URLError

if it's it's full name is urllib.error.URLError, then the declaration will be:

try:
    from urllib2 import urlopen, Request
    from urllib.error import URLError
except ImportError:
    from urllib.request import urlopen, Request, URLError

[ArionMiles]

the line compressedstream = StringIO.StringIO(dat) in your code is invalid for python3 though. from this:

if dat[:2] == '\037\213':
        # Data is gzip encoded, decode it
        compressedstream = StringIO.StringIO(dat)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        extracted_data = gzipper.read()
        dat = extracted_data
        return dat
    return dat #new line

BTW, since you said you don't know much GitHub, I need to send a Pull Request with your code. This tutorial might help.

[dwilbanks]

I'm using python 3. do you mean it's not working in python 2? I have Python 3.5.2 in my system path, which qbittorrent using. I've installed 2.7 separately, but, it can't run nova2.py That does not make any sense to me unless qbittorrent is identifying which version of python it's using and replacing the nova2.py. That would really be weird.

This will return up to 300 results. It will loop through up to 10 requests to the server, and exit out if either it gets no responses from the server or if it calculates it's gotten the last result

I've changed StringIO.StringIO to just StringIO

#VERSION: 1.10
#AUTHOR: Arion_Miles (https://github.com/ArionMiles/)
#LICENSE: MIT License
from xml.dom import minidom
from novaprinter import prettyPrinter
from helpers import download_file 
from io import StringIO
import gzip

try:
    from urllib2 import urlopen, Request, URLError
except ImportError:
    from urllib.request import urlopen, Request, URLError

def retrieve_url_gzip(url):
    """ Return the content of the url page as a string """
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0'}
    req = Request(url, headers = headers)
    try:
        response = urlopen(req)
    except URLError as errno:
        print(" ".join(("Connection error:", str(errno.reason))))
        print(" ".join(("URL:", url)))
        return ""
    dat = response.read()
    # Check if it is gzipped
    if dat[:2] == '\037\213':
        # Data is gzip encoded, decode it
        compressedstream = StringIO(dat)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        extracted_data = gzipper.read()
        dat = extracted_data
        return dat
    return dat #new line

class zooqle(object):
    """ Search engine class """
    url = 'https://zooqle.com'
    name = 'Zooqle'
    supported_categories = {'all'       : 'all',
                            'movies'    : 'Movies',
                            'tv'        : 'TV',
                            'music'     : 'Music',
                            'games'     : 'Games',
                            'anime'     : 'Anime',
                            'software'  : 'Apps',
                            'books'     : 'Books',
                            'others'  : 'Other'}

    def download_torrent(self, info):
        """ Downloader """
        print(download_file(info))

    def search(self, what, cat="all"):
        """ Performs search """
        page = 1
        while page < 11:
            query = "".join((self.url, "/search?q=", what, "+category%3A", self.supported_categories[cat], "&fmt=rss"))
            if( page>1 ):
                query = query + "&pg=" + str (page)
            response = retrieve_url_gzip(query)
            xmldoc = minidom.parseString(response)
            itemlist = xmldoc.getElementsByTagName('item')
            if( len(itemlist ) ==0):
                return
            for item in itemlist:
                zooqle_dict = zooqle_dict = {"engine_url" : self.url}
                zooqle_dict['name'] = item.getElementsByTagName('title')[0].childNodes[0].data
                zooqle_dict["link"] = item.getElementsByTagName('enclosure')[0].attributes['url'].value
                zooqle_dict["desc_link"] = item.getElementsByTagName('link')[0].childNodes[0].data
                zooqle_dict["size"] = item.getElementsByTagName('enclosure')[0].attributes['length'].childNodes[0].data
                zooqle_dict["leech"] = item.getElementsByTagName('torrent:peers')[0].childNodes[0].data
                if not zooqle_dict["leech"].isdigit():
                    zooqle_dict["leech"] = ''
                zooqle_dict["seeds"] = item.getElementsByTagName('torrent:seeds')[0].childNodes[0].data
                if not zooqle_dict["seeds"].isdigit():
                    zooqle_dict["seeds"] = ''
                prettyPrinter(zooqle_dict)
            totalResultVal  = xmldoc.getElementsByTagName('opensearch:totalResults')[0].childNodes[0].nodeValue
            startIndex  = xmldoc.getElementsByTagName('opensearch:startIndex')[0].childNodes[0].nodeValue
            itemsPerPage = xmldoc.getElementsByTagName('opensearch:itemsPerPage')[0].childNodes[0].nodeValue
            if( ( int(startIndex)  + int(itemsPerPage) > int( totalResultVal ))):
                return
            page += 1
        return

[dwilbanks]

When they return a size value of 0, that actually means they don't actually have the torrent, only the meta data.

It should return the magnet link instead.

I've looked at the the guide you linked to, and several others. Since we are no in the 20th century I no longer use command lines to perform tasks that I don't have to.

I also don't use notepad as my text editor. Eventually I'll find a guide of how to use github in the 21st century with modern tools. Until then all my gold won't get shared.

[dwilbanks]

This version give qb the magnet link if it knows that the torrent download link will be invalid based on the observation that torrents that show up as 0 byte don't have valid download links.

For myself, I've got a process that uses the web api to scrape off any hashes that were loaded by magnets and don't have their torrents, then searches alternative sites for them.

#VERSION: 1.10
#AUTHOR: Arion_Miles (https://github.com/ArionMiles/)
#LICENSE: MIT License
from xml.dom import minidom
from novaprinter import prettyPrinter
from helpers import download_file 
from io import StringIO
import gzip

try:
    from urllib2 import urlopen, Request, URLError
except ImportError:
    from urllib.request import urlopen, Request, URLError

def retrieve_url_gzip(url):
    """ Return the content of the url page as a string """
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0'}
    req = Request(url, headers = headers)
    try:
        response = urlopen(req)
    except URLError as errno:
        print(" ".join(("Connection error:", str(errno.reason))))
        print(" ".join(("URL:", url)))
        return ""
    dat = response.read()
    # Check if it is gzipped
    if dat[:2] == '\037\213':
        # Data is gzip encoded, decode it
        compressedstream = StringIO(dat)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        extracted_data = gzipper.read()
        dat = extracted_data
        return dat
    return dat #new line

class zooqle(object):
    """ Search engine class """
    url = 'https://zooqle.com'
    name = 'Zooqle'
    supported_categories = {'all'       : 'all',
                            'movies'    : 'Movies',
                            'tv'        : 'TV',
                            'music'     : 'Music',
                            'games'     : 'Games',
                            'anime'     : 'Anime',
                            'software'  : 'Apps',
                            'books'     : 'Books',
                            'others'  : 'Other'}

    def download_torrent(self, info):
        """ Downloader """
        print(download_file(info))

    def search(self, what, cat="all"):
        """ Performs search """
        page = 1
        while page < 11:
            query = "".join((self.url, "/search?q=", what, "+category%3A", self.supported_categories[cat], "&fmt=rss"))
            if( page>1 ):
                query = query + "&pg=" + str (page)
            response = retrieve_url_gzip(query)
            xmldoc = minidom.parseString(response)
            itemlist = xmldoc.getElementsByTagName('item')
            if( len(itemlist ) ==0):
                return
            for item in itemlist:
                zooqle_dict = zooqle_dict = {"engine_url" : self.url}
                size = item.getElementsByTagName('enclosure')[0].attributes['length'].childNodes[0].data
                zooqle_dict['name'] = item.getElementsByTagName('title')[0].childNodes[0].data
                if(size=="0"):
                    zooqle_dict["link"] = item.getElementsByTagName('torrent:magnetURI')[0].childNodes[0].data
                else:
                    zooqle_dict["link"] = item.getElementsByTagName('enclosure')[0].attributes['url'].value  

                zooqle_dict["desc_link"] = item.getElementsByTagName('link')[0].childNodes[0].data
                zooqle_dict["size"] = size

                zooqle_dict["leech"] = item.getElementsByTagName('torrent:peers')[0].childNodes[0].data
                if not zooqle_dict["leech"].isdigit():
                    zooqle_dict["leech"] = ''
                zooqle_dict["seeds"] = item.getElementsByTagName('torrent:seeds')[0].childNodes[0].data
                if not zooqle_dict["seeds"].isdigit():
                    zooqle_dict["seeds"] = ''
                prettyPrinter(zooqle_dict)
            totalResultVal  = xmldoc.getElementsByTagName('opensearch:totalResults')[0].childNodes[0].nodeValue
            startIndex  = xmldoc.getElementsByTagName('opensearch:startIndex')[0].childNodes[0].nodeValue
            itemsPerPage = xmldoc.getElementsByTagName('opensearch:itemsPerPage')[0].childNodes[0].nodeValue
            if( ( int(startIndex)  + int(itemsPerPage) > int( totalResultVal ))):
                return
            page += 1
        return

[dwilbanks]

in my version of the code, I changed the name of retrieve_url() to retrieve_url_gzip() because I thought that was your purpose, now that I look closer at the version of retrieve_url() they provide I see that you did it because the force decoding entities that you need to continue to be entities.
I guessed wrong about your purpose, but, I guessed correctly that it needed a better name.

[Pireo]

@dwilbanks sorry for the little OT, I just wonder if you can fix this: https://pastebin.com/AS7nC98F it has a similar problem, only 10 results, works with python2 only, and can't click on torrents to see the files neither redirect to the download page.

[dwilbanks]

@Pireo , I looked at https://btdb.in long enough to figure out that I don't want it.

First and most importantly, it has no torrent links, so everything you get will be zero byte waiting for discovery. The magnet links they do give you don't include the full list of tracker, so your discovery might be very slow.

There is no magic way of getting a server that's configured to give you 10 results to give you more than 10 results. For getting more than 10, you'll need to do just what I did in this code. I created a loop, set a maximum number of page reads then generated the URL to read including the page number parameter.

You've got a lot of work ahead of you, for not much value to anyone.

btdb.in is a very aggressive advertising site. They don't care about torrents.

[ArionMiles]

If anyone is able to find a solution for this, send a PR and I'll review and merge it.

[dwilbanks]

Are there new symptoms? I guess now we need to figure out why it's working on my python 3 install and not yours.

[ArionMiles]

It's just that I can't test the code, and since you're not "living in the 20th Century", someone else might need to figure it out and send a PR, which is when I'll work on getting python installation fixed on my side and test it, then merge it.

[dwilbanks]

ah, I see. Copy paste has actually been around much longer. I'm not sure about windows 3.1 but, I'm sure that CTRL-C CTRL-V worked fine in windows 95.

It seems you don't appreciate my help. I'm sorry to have been such a bother to you.

[ArionMiles]

It's just that it's your contribution and you should get proper credit for that. And you get to learn to make PRs, so there's that.

[dwilbanks]

I wanted to have the end result. I now have a working zooqle plugin. I have no interest in getting credit.

I would like to learn how to use github, but, not by using command lines that I will forget an hour after typing them. The linux nerds of the world love typing long strings into the command line that must be letter perfect.

I use an IDE for development (Eclipse). It may not be the best, and if there are better, I'm willing to change. If I do change to a better IDE, it will need to be better than eclipse, all around.

Eclipse has direct support for github, except I don't know how do use it. I suspect that it requires a lot of command line setup to make it actually work.

[ArionMiles]

This issue is fixed in latest version. I got it to work on my Linux partition using python3, but still can't make it work on Windows, so there's a issue on my end.