search

qbittorrent / search-plugins

Search plugins for the search feature
GNU General Public License v2.0
4.32k stars 478 forks source link

Discussion: Breaking changes in qBittorrent search-engine #121

Open ngosang opened 4 years ago

ngosang commented 4 years ago

Update: Proposal => qBittorrent.pdf

Hello everybody,

TLDR: I don't have time to maintain the plugins, the search-engine code needs a huge refactor. I'm proposing to remove all search plugins and Python code and make a native integration with https://github.com/Jackett/Jackett

First of all, I think this feature is used by thousands (take a look at Reddit) and qBittorrent is the only bittorrent client with this functionality. So, I don't want to remove it. Furthermore, in the latest releases is supported in the WebUI making it convenient for seedboxes.

The current implementation (code) is a mess, both, the C++ and Python parts. In this issue https://github.com/qbittorrent/search-plugins/issues/84 I list all required changes to make it clean and more maintainable. That are a lot of changes, breaks compatibility, I don't have the skills to make the changes in the C++ code, and more important, there aren't maintainers for the plugins.

Instead of doing the changes, I propose to remove all Python code, including official and unofficial plugins and make native integration with Jackett for the following reasons.

I want to know what do you think about and I would like you to try Jackett https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin

ping @qbittorrent/demigods @qbittorrent/webdev @qbittorrent/frequent-contributors @qbittorrent/bug-handlers

UPDATE: Implementation draft

NOTE: In the first version I will try to make as few changes as possible.

1. Enable search engine

2. Install plugins / search configuration

3. Search UI

4. Perform searches with Torznab

5. Future steps

glassez commented 4 years ago

We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.

👍

Piccirello commented 4 years ago

I like it, but I'm already a Jackett user.

ngosang commented 4 years ago

Updated the first message with the changes required.

@Chocobo1 Would you like to work in the development? I can test everything carefully. :)

Chocobo1 commented 4 years ago

@Chocobo1 Would you like to work in the development? I can test everything carefully. :)

I probably won't have much time to do anything significant.

ngosang commented 4 years ago

If you don't mind I can help.

Of course you can. You have all the details in the first message. Just mention me when you have a PR or something. If at any point you are not longer interested in implementing this, please, notice me.

ghost commented 4 years ago

Imho the python integration has been there for a long time and it would not be ideal to drop it completely after all the work it required, just drop python2. Also, not all sites are supported and you can't use the "go to the description page" option.

hannsen commented 4 years ago

These would be the sites that we have but jackett does not:

Academic Torrents
ali213.net
anidex.info
CineCalidad
HorribleSubs
Linux Tracker
small-games.info
UnionDHT

But yeah, it sounds like a sensible idea, and there is no reason why these sites above could not be implemented in jackett.

and you can't use the "go to the description page" option.

Jackett does support that

ngosang commented 4 years ago

I opened some requests in Jackett to add them. anidex.info, HorribleSubs are in Jackett already. @hannsen Maybe you can help porting some of them to Jackett, or just helping Jackett somehow. :)

jarmo commented 4 years ago

Has anything already been done regarding this? I did upgrade my qBittorrent to 4.2.3 after which I still see all the plugins I had before, but "categories" only list "All categories". Also, sometimes plugins do not find anything anymore.

I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).

Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).

ngosang commented 4 years ago

As far as I know no one is working in this. Most of the plugins included in qBittorrent are working (just make sure you have them updated). The categories are not tested well, just use "all categories".

I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).

I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.

Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).

I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections. PRs are welcome but I don't see any.

jarmo commented 4 years ago

@ngosang thank you for your thoughts.

The categories are not tested well, just use "all categories".

My point was that they were working before and stopped working after upgrading. That's why I asked. It doesn't really matter to me because I always have been using "all categories" anyway. Just an observation.

I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.

Looking at the Jackett logs that was my initial suspicion too, that it works exactly like you just described in here. Good that you confirmed it, since that explains a lot. When I did enable all (public) indexers in the beginning then I had to start looking at the log and disable some, which were failing thus causing entire search to not return any results. Using your proposed API does make sense and sounds like a good idea.

I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections.

I totally understand this. One way of solving this problem would be to create better integration between Jackett and qBittorrent where Jackett is bundled with qBittorrent or can be installed somehow throught qBittorrent and it would start/stop Jackett server and configure API key automatically or whatever so that end-user would not need to know anything about Jackett or how to configure it etc. I'm quite sure that this is also not a small task and I totally understand that PRs are welcome since I'm developing some OSS stuff too.

Just my thoughts how things could be improved.

cyberfunk commented 4 years ago

...> Just my thoughts how things could be improved.

Can you tell me which version of Qbittorent has working search for you? I'm talking about the regular search plugins not Jackett (Which I've never been able to get working) I'm on latest 4.2.5 but search has been broken for a long time now. All I get is a Yellow triangle with exclamation point. All search plugin URL's work in browser but do not work in Qbittorent. I tried both with proxy enabled and disabled. Nothing. I'm on a Mac running Catalina with python 3.7.7 installed.

ngosang commented 4 years ago

The search feature is working in all versions of qBittorrent. A small number of users have problems because qBittorent can't find Python (it's an external program and we don't have control). Python 3.7.7 is supported too, just take a look at the qBittorrent log to be sure it's detecting the right Python version. Some users have several version of Python in the same machine and that can cause problems too.

cyberfunk commented 4 years ago

Here is my latest log.

1
ngosang commented 4 years ago

That looks good. Could be a lot of things, even the torrent sites can be blocked in your country...I can't help much more. I recommend you Jackett by the way.

cyberfunk commented 4 years ago

According to the log Python 3.7.7 is being detected. When I execute a search it just fails immediately with a yellow triangle and exclamation point but the log shows nothing regarding my failed searches. Is there another place to look for these failure logs other than the Execution log tab?

ngosang commented 4 years ago

No sorry, you are blind. The search functionality is not used by most developers and they don't care about the search log.

cyberfunk commented 4 years ago

The torrent sites are all up and running beautifully when I type each URL manually in my Safari browser. It's Qbittorent search that fails even when the URLS are working perfectly.

cyberfunk commented 4 years ago

Are you using a Mac or are you on Windows or Linux?

ngosang commented 4 years ago

Windows and Linux. Definitely could be a Mac thing because we have few users.

dusan985 commented 4 years ago

Hello everybody.

For me, Jackett plugin in qBittorrent has been working excellent up until today. Jackett, plugin and qBittorrent have been updated to the latest versions. I've tried a search and it seems that the results cannot be parsed back into qBittorrent search window (log below). Any suggestions?

2020-05-04 09:40:16 Error System.NullReferenceException: Object reference not set to an instance of an object. at Jackett.Common.Models.ResultPage.b__13_0(ReleaseInfo r) in d:\a\1\s\src\Jackett.Common\Models\ResultPage.cs:line 62 at System.Linq.Enumerable.SelectListIterator`2.MoveNext() at System.Xml.Linq.XContainer.AddContentSkipNotify(Object content) at System.Xml.Linq.XContainer.AddContentSkipNotify(Object content) at System.Xml.Linq.XElement..ctor(XName name, Object[] content) at Jackett.Common.Models.ResultPage.ToXml(Uri selfAtom) in d:\a\1\s\src\Jackett.Common\Models\ResultPage.cs:line 38 at Jackett.Server.Controllers.ResultsController.Torznab(TorznabRequest request) in d:\a\1\s\src\Jackett.Server\Controllers\ResultsController.cs:line 420
2020-05-04 09:40:16 Info Found 3128 releases from AggregateSearch for: westworld
cyberfunk commented 4 years ago

Windows and Linux. Definitely could be a Mac thing because we have few users.

Thank you for providing that information. If anybody reading this is on a Mac with Jackett and native search plugins working please let me know version number as I have not had working search for many releases.

cyberfunk commented 4 years ago

I think I solved my issue. It was my socks5 proxy that was blocking search. I did a fresh install and deleted all preferences and search works:-)

dracossan commented 4 years ago

Hello,

quick question about Jackett. Well configure , i can get answer from jacket to QBT. no issue.

but seems there is a limit in term of item received. the search plugin from qbt return more item than the jacket one ( by using the same web site ). i can't get more than 60 item per indexer from jackett which ever indexer i use.

any idea ?

Dycast commented 4 years ago

I've tried numerous times to update, but am having problems with UnionDHT - It returns results, but clicking on the result does not 'fire off' the torrent to download, I can go to the description page, but it seems no matter what I try I cannot get the torrent to start downloading?

Otherwise I've turned a few others off but have most of them working, ones I turned off were because returning jibberish (or maybe diff language?) or found everything was in a diff. language on the search,

I agree Jackett is neat, but was kinda a hassle to setup, and since has broken and I've just left it..

The only thing I would like to see would be to show new plugins available without having to manually search the plugin directory. Like from the 'update plugins' if it could search the directory and show new ones available that would be great! I know some of the sub only ones may not be able to do that with.

ngosang commented 4 years ago

UnionDHT is unofficial. Reach the original author => https://github.com/nindogo/qbtSearchScripts

ghost commented 3 years ago

@ngosang to be honest, the official search engine plugins are perfect with the exception of EZTV imho, which I think doesn't deserve to be included. Let it be unofficial. It's fake, has illegal stuff and not generic. @nindogo if you want to replace it with something else then I recommend TorrentGalaxy which made it into the top 10 this year: https://torrentfreak.com/top-torrent-sites-2021-210103/ but I'm fine with 9 sites.

polarathene commented 3 years ago

Installed a newer release of qBittorrent on a Windows machine the other week and Jackett was not available (previous installed version did not have support for Jackett IIRC). Presumably due to the previous install? Ended up going on to github, locating the plugin source and copy/pasting in plugin directory.

Just a heads up, that existing users may not get Jackett support when updating (not that it matters too much since they've still got to manually create a simple config too, unless the feature shown here arrives).

Is that going to be an issue if Jackett becomes the only option? Will the others disappear or not work, while the wiki page instructs that Jackett would be available by installing qBittorrent? May need a warning/notice to users when Jackett isn't installed/detected.

glassez commented 3 years ago

In fact, we have the Search subsystem in an unmaintainable state. The idea of getting rid of it in favor of implementing the support of the Jackett also looks unsuccessful, since firstly there are no people willing to implement it, and secondly it will replace one external dependency with another, which will require taking care of it throughout the entire life cycle (and this in turn will sooner or later lead it to the same state as the current one). We (core developers) are concerned about this problem and are looking for compromise solutions. Recently, we have been actively discussing the idea of redesigning the Search subsystem in such a way as to make it maintenance-free (to get rid of all direct dependencies and all responsibility for downloading, installing and updating plugins, even from the very essence of plugins).

Here is its concept:

  1. Drop direct support of "nova" and its plugins (including any cares for Python);
  2. Define a communication protocol (just a set of conventions about the format of command-line parameters and returned data) for interacting with so-called "search providers". The search provider can be any executable that supports the previously mentioned communication protocol.
  3. Implement a method for registering search providers in the app and the UI for it.

From the above, it can be seen that once implemented and debugged, the new Search subsystem will no longer require almost any maintenance on our part. But at the same time, third parties will be free to provide "search providers" with any degree of convenience, whether it is just an executable file of the provider that you need to download and register manually, or an installer that will install and register the provider and its dependencies (or several providers) for you. Providers can also be distributed through various package managers (apt etc.), which will allow you to update them in the same way as any other application/library. And given that the provider can be any executable file (not even a file, but anything that can act as an executable command), the provider can be written in any programming language that you prefer. In order not to upset existing users, I note that the transition process should not be too burdensome for them. We can take care of converting the installed "nova" plugins to a compatible format and register them automatically (and then let them float freely).

Comments are welcome. But do not forget that we are talking about choosing between an actually unmaintained feature and maintenance-free one.

polarathene commented 3 years ago

In order not to upset existing users, I note that the transition process should not be too burdensome for them. We can take care of converting the installed "nova" plugins to a compatible format and register them automatically (and then let them float freely).

That was my main concern. Would not be great to update one day and find the search that was previously fine was no longer available/supported, eg Jackett support.

Implement a method for registering search providers in the app and the UI for it.

Each of these search provider executables could have a command to provide some config info which could be used for registering, perhaps a JSON/TOML file. The app stores a copy of that data and presents it in the UI? Is it possible to fall out of date if the search provider updates? eg if version info was included, or would you query/update this periodically like on restart of the app?

If they were instead running as small REST API servers, they could also receive a standard JSON file adhering to some schema that can be versioned that contains query parameters, or is this going to be split into several option flags for executables? (could alternatively stringify a JSON or similar serialized format?)

glassez commented 3 years ago

@polarathene You don't seem to understand the concept... Everything should be much easier than you think/suggest. All that is required of "search provider" is to accept certain command-line parameters, do its job, and then print the results in stdout using a predefined format (pretty much the same thing "nova" plugins doing now).

Is it possible to fall out of date if the search provider updates? eg if version info was included, or would you query/update this periodically like on restart of the app?

No additional capabilities, including taking care of updating providers, are supposed to be in the app. (Re-read my previous comment.)

Each of these search provider executables could have a command to provide some config info which could be used for registering, perhaps a JSON/TOML file. The app stores a copy of that data and presents it in the UI?

No. It was about having some sort of configuration file containing occurrences for registered providers (just a display name and a command line to start the provider, maybe something else). Perhaps it will not be a single file, but one file per provider, so that external installers (package managers) can more easily register the installed providers. The UI simply shows a list of registered providers and allows you to add, edit, and delete its items.

ngosang commented 3 years ago

@glassez I know the search engine is unmaintainable as is but I think you are being too radical. I know you are not using that feature but I guarantee you that it's used by thousands and it's one of the main reasons for the users to move from Transmission/Deluge/uTorrent to qBittorrent. The current search engine is working, does not need too many maintenance and there are a lot of 3rd party plugins maintained. There is no need to make radical decisions. => https://github.com/qbittorrent/search-plugins/wiki/Unofficial-search-plugins

I'm opposed to your solution because:

Maybe it's not clear in the first post but when I say Jackett I mean Torznab, I don't want to be coupled to a single app. I still think the native implementation of Torznab/Jackett REST API is the best idea to decouple things. You will be able to remove most of the search-engine code including all Python related things. You only have to make HTTP calls, parse XML responses. Torznab specification is a standard used by several apps, not just Jackett, and the definition has not changed in more than a decade. Btw, I'm active maintainer of Jackett and I can guarantee the Torznab API won't be changed in years.

Torznab Spec => https://torznab.github.io/spec-1.3-draft/external/newznab/api.html Torznab Spec 2 => https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/

Torznab Spec is used by well known software:

Torznab support a lot of features like categories, IMDB search, private trackers tags (free leech, double upload...). The idea is to make a basic Torznab implementation, that could be extended in the future, and remove existing search-egine code (bump major version). Basic implementation:

  1. Dialog for configuring indexers 1 by 1. User has to provide Tracker name, Torznab URL and API Key. When user press save, you can do a basic request (search without parameters) to test if it's working.
  2. The search dialog will be the same. If you want to simplify the installation more, you can remove the categories but since they are supported by Torznab and they are similar it's easy to implement.
  3. When the user click "search" you have to make 1 HTTP request per indexer in parallel and wait for the responses. The URL is provided by the user, you only have to fill some query parameters: API key (provided by the user), query string and category. The responses are always in XML. Check if the HTTP response code is 200. Map the XML to an Object/Class (it will be always the same for all indexers). Add the torrents to the list. Show an error in the logs or somewhere if some indexer fails and keep going with the others.
  4. That's all. As I said Torznab supports many search parameters and the response contains more fields that we are not using at this moment like torrent categories, torrent hash (we can group results in the future).... This could be implemented in the future by adding more logic in qBittorrent.

If you don't agree with that I can suggest other changes but that will be more work and they won't make you really happy.

@qbittorrent/bug-handlers @sledgehammer999 more feedback is welcome. If anyone is interested in working on this I can provide support, examples (request responses), even access to one of my servers with a configured Jackett instance. I can test the PRs and finish with remaining taks: deprecate this repository, remove Python from qBittorrent Docker image...

FranciscoPombal commented 3 years ago

@ngosang

I still think the native implementation of Torznab/Jackett REST API is the best idea to decouple things. You will be able to remove most of the search-engine code including all Python related things. You only have to make HTTP calls, parse XML responses.

:+1: I would love it if qBittorrent completely dropped Python. From my experience with the issue tracker, it is the source of many issues, especially on Windows, and especially for users who have several versions installed. The "Python detection" code is simply not doing its job, and it's a hassle to maintain, especially once you account for the numerous ways to install Python there are, the different Python distributions, etc.

I also recognize the fact that the Search engine is one of, if not the best value proposition that qBittorrent brings to the table for the average user. It's a huge let down for them whenever it doesn't work properly.

Additionally, I too can confirm that Jackett and friends are "the standard" when it comes to tools for torrent search. It's mature and will be around a long time. If we support it well, users have no reason to complain to us, they can direct any feedback upstream.

qBittorrent already has an HTTP server implementation, that is probably good enough already for the purpose you describe. If not, it could probably be brought up to a good enough state relatively easily, but @glassez can probably speak with more authority and in more detail about this.

As for parsing XML: that's a bummer. It basically means we'll have to adopt another 3rd party dependency (I hope nobody will suggest writing and maintaining our own XML parser). Not the end of the world, but annoying. Is it not possible to have JSON as the data interchange format here? That way, we could use Boost.JSON, which, as opposed to an entirely new library from a completely different source, is just another Boost component, from a "family of dependencies", that we already heavily depend on anyway (directly for a few things, and transitively through libtorrent, which makes heavy use of Boost libs).

glassez commented 3 years ago

As for parsing XML: that's a bummer. It basically means we'll have to adopt another 3rd party dependency (I hope nobody will suggest writing and maintaining our own XML parser).

That way, we could use Boost.JSON, which, as opposed to an entirely new library from a completely different source

You seem to be out of topic... Qt provides both XML and JSON parsers (e.g., the first is used in RSS subsystem whilst the second one is widely used across several components of qBittorrent). So it isn't a problem here.

glassez commented 3 years ago

qBittorrent already has an HTTP server implementation, that is probably good enough already for the purpose you describe.

Is some built-in HTTP server needed to implement Jackett support?

Additionally, I too can confirm that Jackett and friends are "the standard" when it comes to tools for torrent search. It's mature and will be around a long time. If we support it well, users have no reason to complain to us, they can direct any feedback upstream.

I'm afraid that by getting rid of the need to maintain one third-party app (Python), we'll get another "pain in the ass", in the form of a lot of requests from (lazy) users to have qBittorrent maintain yet another third-party app (Jackett).

glassez commented 3 years ago

After briefly reviewing the code of the qBittorrent official Jackett plugin, I see that it doesn't even provide some of the possible values (size, seeds, leeches). Are these the limitations of the Jacket itself or just the plug-in? If the Jacket does not support receiving this information, will it not be another reason for reproaches in our direction if we switch to it?

glassez commented 3 years ago

The idea is to make a basic Torznab implementation, that could be extended in the future, and remove existing search-egine code

@ngosang Something like what an existing Jackett plugin does would be enough?

glassez commented 3 years ago

Torznab Spec => https://torznab.github.io/spec-1.3-draft/external/newznab/api.html

It says that I can optionally get data in the form of JSON. Does it really work in Jackett?

FranciscoPombal commented 3 years ago

@glassez

You seem to be out of topic... Qt provides both XML and JSON parsers (e.g., the first is used in RSS subsystem whilst the second one is widely used across several components of qBittorrent). So it isn't a problem here.

😅 You're right of course :) Good thing this won't be an issue then.

Is some built-in HTTP server needed to implement Jackett support?

Again, a little bit of confusion from my part. Seems we just need to make a few requests using Jackett's API, i.e., we only ever act as a client, which is even simpler.

I'm afraid that by getting rid of the need to maintain one third-party app (Python), we'll get another "pain in the ass", in the form of a lot of requests from (lazy) users to have qBittorrent maintain yet another third-party app (Jackett).

My understanding (and hope) is that the maintenance burden will be minimal for us.

We "just" have to implement the required API calls, and process the received data and display it nicely in the UI. After that we can basically forget about it, support for new plugins is entirely the responsibity of Jackett. Of course, the maintenance burden being minimal for us hinges entirely on the premise that Jackett's API is mature and stable, but @ngosang is saying this is the case.

In fact, since apparently Torznab/Jackett's API is actually a "de facto" specification/protocol at this point, not just some internal one-off thing from some very particular 3rd party program, we could even think about ditching the search plugin concept entirely and just say qBittorrent implements support for the specification as a self-contained subsystem that is part of the core (I think this is exactly what @ngosang is suggesting, just worded a little differently) - i.e., we accept data from any program that sends it in the expected format, conformant to the spec.

This would be much like the relationship that qbittorrent has with RSS - qbittorrent implements the necessary bits of the protocol/spec so that the user can add any valid feed; there is no community-maintained repository of "RSS plugins" for each individual site. Of course, unlike the Torznab/Jacket API, I beleive RSS is standardized with an actual RFC (or multiple?), which is a stronger guarantee, but I believe the point still stands in spite of that.

glassez commented 3 years ago

we accept data from any program that sends it in the expected format

This is one of the key aspects of my concept as well.

I'm afraid that by getting rid of the need to maintain one third-party app (Python), we'll get another "pain in the ass", in the form of a lot of requests from (lazy) users to have qBittorrent maintain yet another third-party app (Jackett).

My understanding (and hope) is that the maintenance burden will be minimal for us.

I insist that we are no longer responsible for installing any third-party applications. If we do support Jackett, the responsibility for installing and configuring it should be entirely on the user himself. All qBittorrent should care about is maintaining the parameters it needs to interact with Jackett.

Offtopic:

I beleive RSS is standardized with an actual RFC (or multiple?), which is a stronger guarantee

I hope that Torznab/Jackett's API is something more strict/reliable than the RSS. Personally, I can't bring myself to call RSS a standard. It looks like a bunch of "may", "recommended" , etc. In addition, even what is defined there more or less strictly, is not respected by most of the so-called "web developers" who implement it on their sites. Because of this, our "generic" RSS support has a bunch of issues with supporting certain feeds.

Really, I'm not so strictly against implementing Jackett support instead of the existing subsystem. This looks like a worthwhile compromise. I am waiting for answers to my comments/questions above from interested parties (especially from @ngosang). Perhaps I will do this job if we agree on the key aspects of the problem.

ngosang commented 3 years ago

Is some built-in HTTP server needed to implement Jackett support?

No. You only need to make HTTP requests to a remote service and parse the response. Jackett is stateless so you don't have to keep sessions or other things between requests neither. It's really simple.

It says that I can optionally get data in the form of JSON. Does it really work in Jackett?

The response is standard XML and can be parsed with Qt. The specification mentions JSON too but it's not implemented in Jackett. I never saw other Torznab programs using JSON, always XML.

I'm afraid that by getting rid of the need to maintain one third-party app (Python), we'll get another "pain in the ass", in the form of a lot of requests from (lazy) users to have qBittorrent maintain yet another third-party app (Jackett).

Don't worry about that. As I said, I'm maintainer and we have more than 20k daily downloads. There are other projects like Cardinann that follow the same Torznab specs. Jackett is not going anywhere, too many projects depend on it.

After briefly reviewing the code of the qBittorrent official Jackett plugin, I see that it doesn't even provide some of the possible values (size, seeds, leeches). Are these the limitations of the Jacket itself or just the plug-in? If the Jacket does not support receiving this information, will it not be another reason for reproaches in our direction if we switch to it?

It provides that information and more => https://github.com/qbittorrent/search-plugins/blob/master/nova3/engines/jackett.py#L136 The thing is Jackett supports 500 different torrents sites and not all the fields are available in all the sites. Some fields like the torrent name and download link are always present but other fields like the size, seeds, leechs can be null/unknown. We have to deal with that but we are already doing that in the current search engine.

Something like what an existing Jackett plugin does would be enough?

Yes and no. The basic functionality is covered by the plugin. Search request + XML response parsing. The thing is the plugin is cheating a bit because it only makes 1 request. You can configure several indexers/sites in Jackett. The right thing to do is to make 1 Torznab request to each indexer. Instead of that, there is a hidden "indexer" in Jackett called "all" that internally calls all configured indexers and return 1 huge response after a lot of time. That endpoint cases problems if some of the indexers fails. It takes a long time until the endpoint responses. There is a limit in the number of results in the response... I would like to allow configuring indexers 1 by 1 in qBittorrent and make requests in parallel. I know this is more work but the solution will be much better and forever.

@glassez I elaborated a document with the details. Take your time. qBittorrent.pdf

glassez commented 3 years ago

It seems that the concept of using Torznab differs from mine only in the protocol, doesn't it?

@ngosang In your Indexer configuration example you have both API path and Indexer URL fields. Seems API path is redundant since URL contains it. Or am I missing something?

ngosang commented 3 years ago

In your Indexer configuration example you have both API path and Indexer URL fields. Seems API path is redundant since URL contains it. Or am I missing something?

The Torznab URL that you can copy & paste from Jackett UI doesn't have the /api part. imagen

In other software like Radarr or Sonarr the /api part is set by default but it can be changed by the user. I don't know if there are other implementations out there with different terminations.

The Torznab specs one uses /api in the examples and the other doesn't mention.

For now, I think we can assume the URL has to end with /api (we have to concat that part if it's not in the URL). That way we can save one field. It can be added in the future is someone complains.

FranciscoPombal commented 3 years ago

@ngosang

Cheers for the document, really nice.

I have 2 alternative proposals concerning "indexer management":

1.

Is it possible to obtain a list of all possible indexers by querying the API? If so, qBittorrent could automatically make such a request and show all available indexers to the user. This also allows auto-filling all of the 4 mandatory text fields except the API key (page 4, 2nd figure), assuming the API returns a usable Name. Then, to get an indexer to work, it would have to be both configured and enabled (there can be a column for each property). If an indexer is not configured, it shows up as greyed-out in the list, and when selected, the only available button for clicking is a "configure indexer" button. Once an indexer is configured, it can either be enabled or disabled as normal.

2.

Maybe indexer management could be a tab instead of a dialog? Not a very important decision, but worth thinking about.

Also, you mention that querying all indexers is error-prone and incurs a performance penalty. We should probably warn the user (with a red-lettered label or something) that executing searches with many indexers enabled may take a long time.

Another note: It should be possible to search with only a subset of enabled indexers, not just one or "all enabled" (but that capability can probably be left out of a first implementation).

glassez commented 3 years ago

Well, I'm going to do this job in the meantime.

Maybe indexer management could be a tab instead of a dialog? Not a very important decision, but worth thinking about.

  1. Why?
  2. Do you mean the tab in the same line with regular "search result" tabs?

In any case, it will initially be a dialog, since the other option is controversial.

Is it possible to obtain a list of all possible indexers by querying the API? If so, qBittorrent could automatically make such a request and show all available indexers to the user. [...]

I still insist on implementing it to be configurable at "independent indexers" basis I still insist that it can be configured based on "independent indexers", so we don't have a hard dependency on the Jackett or anything else. Just a set of individual endpoints that support Torznab protocol. Of course, we can later add this feature as an additional action "Get a list of indexers from a URL".

FranciscoPombal commented 3 years ago

@glassez

  1. Why?

  2. Do you mean the tab in the same line with regular "search result" tabs?

  1. I would prefer it to be a non-modal UI element, so that interaction with it is more seamless.

  2. I mean as "subtabs" within the search tab itself. A picture is worth a thousand words (imagine both buttons with red text switch between "subtabs"; here, the results "subtab" would be active):

    screenshot

Of course, I don't mind too much if you do end up implementing it as a dialog, this is simply a suggestion.

I agree with the rest of your post.

glassez commented 3 years ago

I mean as "subtabs" within the search tab itself. A picture is worth a thousand words (imagine both buttons with red text switch between "subtabs"; here, the results "subtab" would be active):

Well, it looks acceptable, except that it loses a bit of vertical space, which might worry someone.

FranciscoPombal commented 3 years ago

@glassez

Well, it looks acceptable, except that it loses a bit of vertical space, which might worry someone.

Good point, agree that horizontal space is cheaper. But the button to bring out the dialog would take up approximately the same amount of space. So I guess the ideal solution would be to position the tabs vertically on the right, the same we do it for the banned IPs tab.

ngosang commented 3 years ago

Well, I'm going to do this job in the meantime.

Nice to hear that. Please, quote me when there is something to test. I can help with that.

glassez commented 3 years ago

Torznab Spec => https://torznab.github.io/spec-1.3-draft/external/newznab/api.html

In the docs above is said:

The returned XML data stream is RSS 2.0 compliant and also contains additional information in the extra namespace.

But actually I see Jackett provide rss version="1.0" (although the content seems to look like RSS 2.0). @ngosang, can you explain, what happens?