Discussion: Breaking changes in qBittorrent search-engine

ngosang commented 4 years ago

Update: Proposal => qBittorrent.pdf

Hello everybody,

TLDR: I don't have time to maintain the plugins, the search-engine code needs a huge refactor. I'm proposing to remove all search plugins and Python code and make a native integration with https://github.com/Jackett/Jackett

First of all, I think this feature is used by thousands (take a look at Reddit) and qBittorrent is the only bittorrent client with this functionality. So, I don't want to remove it. Furthermore, in the latest releases is supported in the WebUI making it convenient for seedboxes.

The current implementation (code) is a mess, both, the C++ and Python parts. In this issue https://github.com/qbittorrent/search-plugins/issues/84 I list all required changes to make it clean and more maintainable. That are a lot of changes, breaks compatibility, I don't have the skills to make the changes in the C++ code, and more important, there aren't maintainers for the plugins.

Instead of doing the changes, I propose to remove all Python code, including official and unofficial plugins and make native integration with Jackett for the following reasons.

https://github.com/Jackett/Jackett is a free and open source app that support more than 400 torrent sites. All official and unofficial plugins are supported. They are able to bypass captchas, cloudflare... We can't compete
It's well maintained. I'm contributor and it provides releases every day. We don't have maintainers to update the plugins frequently and most unofficial plugins are not working. https://github.com/Jackett/Jackett/releases
They provide a standard REST API based in Torznab. In simple words, it's a XML format well defined (never changes) and It's used by Radarr, Sonarr and many more software. https://github.com/Sonarr/Sonarr/wiki/Implementing-a-Torznab-indexer In case Jackett is discontinued, there are other projects that support Torznab and will be compatible also (see https://github.com/cardigann/cardigann). I'm talking about Jackett but we can make it generic to support all Torznab clients. Will be easy to request and parse the XML in C++ and keep the current UI/WebUI
We already support Jackett and I know many people using it. They won't have to make big changes. https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin
Runs in the same platforms as qBittorrent does (win, linux, mac)
We won't have external dependencies, Python.
We won't have to worry about updating plugins nor resolving issues.
We won't have legal problems.
We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.

I want to know what do you think about and I would like you to try Jackett https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin

ping @qbittorrent/demigods @qbittorrent/webdev @qbittorrent/frequent-contributors @qbittorrent/bug-handlers

UPDATE: Implementation draft

NOTE: In the first version I will try to make as few changes as possible.

1. Enable search engine

Python is not required as external dependency. We can remove all code related to check Python version, download python, copy python code...

2. Install plugins / search configuration

In a clean install, the list will be empty because we don't provide search engines anymore.
Remove strikethrough parts in the screenshot
When you click "add new indexer" or you "edit one" you will see a modal like this. We should add placeholders, tooltips or something to explain each field.
All fields are mandatory and can be obtained from Jackett UI

3. Search UI

By now we can let the UI as it is. Jackett provides additional information that we can add in the future.

4. Perform searches with Torznab

GET api?apikey=&q=
Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu
Eg response XML => https://pastebin.com/kK48jqAc
Each item is a torrent. The field mapping with the current UI is: qBittorrent column => Torznab name => title size => size (in bytes) seeders => seeders leechers => peers - seeders search engine => torznab indexer name configured by the user (the real name of the indexer isin the response but it can confuse the user) description page => comments download link => link (can be a maget or a http link)
Jackett also support categories. You must use this mapping (several categories are supported separated by comma) => https://github.com/qbittorrent/search-plugins/blob/master/nova3/engines/jackett.py#L64
Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu&cat=2000,5000

5. Future steps

Add more columns in search results
Move search engine to preferences ??
Allow to select several categories in the same search (eg, tv and movies)
Add all torznab categories. There are a lot.
Group search results by infohash. Torznab provides a field with this information

ngosang commented 3 years ago

Yes, you are right. I will fix it in Jackett today. How it affects you?

glassez commented 3 years ago

Yes, you are right. I will fix it in Jackett today.

👍

How it affects you?

I needed to know what format the data is provided in Torznab in order to parse it. When I saw rss version="1.0", I thought, "What the hell is this? What document type is it?". Then I found in Torznab specification that it should be RSS 2.0 (only). I looked closer in XML produced by Jackett, and realized that it was very similar to RSS 2.0, but this declaration still confused me. If it was just a mistake, then it's okay. Just fix it.

I have a sad experience of working with RSS, when even very soft requirements of the standard are often not met by server side. So in this situation, I would prefer that the Jackett bugs, if found, be fixed in Jackett without any workarounds in qBittorrent.

ngosang commented 3 years ago

@glassez it's already fixed. New Jackett version will be released tomorrow. If you find more things just tell me. https://github.com/Jackett/Jackett/pull/11871

Released https://github.com/Jackett/Jackett/releases/tag/v0.18.255

ghost commented 3 years ago

What happened to 1337x? It's not included in the official list anymore. sa3dany made a plugin that works anyway, you may add it? Ah, I noticed EZTV always displays "-1" as leechers.

glassez commented 3 years ago

Well, I've finally completed the bulk of the work on this (https://github.com/qbittorrent/qBittorrent/pull/15126). It's still a draft, since some little things are missing, such as validating the input data, etc. But you can already try it in practice. At the same time, I would have heard preliminary feedback. Just please don't ask for anything more advanced than what is there. I would like to limit myself to the basic functionality at the first stage. Advanced features can be added later.

glassez commented 3 years ago

I repeat https://github.com/qbittorrent/qBittorrent/pull/15126#issuecomment-903713355 here. Maybe I'll get an answer after all...

Recently, I began to worry about one question. What is the real benefit of integrating the search engine system (of course, I don't mean the old subsystem, but the new one). What is there that can't be done just by having qBittorrent (w/o any Search Engine Integration) and Jackett? Maybe this question sounds stupid for those who actively use this feature. But I am not an active user of it, so I would like to get an explanation.

FranciscoPombal commented 3 years ago

@glassez https://github.com/qbittorrent/qBittorrent/pull/15126#issuecomment-944522853

ghost commented 1 year ago

Hello all,

What is the state of the proposed changes on macOS version of QBT 4.5.0 / LT2.0?

According to the WIKI page for Search Plugins this has already been implemented yet I am seeing a nova3 directory wasn't removed upon install of v4.5.0 and I still see Python plugins. No mention of adding a torznab indexer. Do we need to remove the nova3 directory for the new torznab feature to be implemented properly?

ngosang commented 1 year ago

It's not implemented yet and nobody is working on it. The development is here => https://github.com/qbittorrent/qBittorrent/pull/15126 For now we are still using Official / Unofficial plugins written in Python. I recommend you to use the Jackett plugin for those indexers not supported => https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin

amamelia commented 11 months ago

@ngosang I had some issues with Jackett (since resolved) and that led me to discover Prowlarr. I saw that Torznab implementation has stalled, but since Jackett already works I thought that means qBt can talk Torznab well enough already and I could just feed it Prowlarr's info in jackett.json since it's just another Torznab implementation. It failed. I would like to know if it's because qBittorrent has been modified to work specifically with Jackett and nothing else. Or if it's a stupid error on my end.

ngosang commented 10 months ago

@amamelia Jacket is working fine. The qBittorrent plugin only works with Jackett but it could be modified to work with Prowlarr.

Myth1-1 commented 5 months ago

QB最新版“动漫花园”导入不了

travibe commented 1 month ago

As per -> https://github.com/qbittorrent/search-plugins/issues/121#issue-536653344

What is the status of all this? version 5 is out now and there have been major issues with the search plugins disappearing/not working. Are we only using Jackkett now? I think the community would really appreciate some very clear language on what the proper usage of Qbittorrent's search function looks like.

Balls0fSteel commented 1 month ago

@travibe The search plugin disappearing / not working has nothing to do with version 5. The affected users either had very old Python runtimes or unmaintained, outdated plugins installed. The plugins auto install, auto update and just work out of the box, just like they did before as of today.

If you require assistance with your installation, please visit our Reddit, Discord or Forums, whichever you prefer.

travibe commented 1 month ago

@travibe The search plugin disappearing / not working has nothing to do with version 5. The affected users either had very old Python runtimes or unmaintained, outdated plugins installed. The plugins auto install, auto update and just work out of the box, just like they did before as of today.

If you require assistance with your installation, please visit our Reddit, Discord or Forums, whichever you prefer.

Thank you, I greatly appreciate you following up on this, and for being so quick as well.

xavier2k6 commented 1 month ago

@Balls0fSteel @travibe

https://github.com/qbittorrent/qBittorrent/issues/21579#issuecomment-2406910742

p060477 commented 1 month ago

my main question is: should be the new qbittorrent vers. also still PORTABLE...???...i mean that the change you make in the search engines feature will be compatible with the portability of Qbittorrent and Jackett,i've also it in portable way, cause my main aim is to have as more portable sw on my pc as i can... thxs so much in adv cheers

qbittorrent / search-plugins

Discussion: Breaking changes in qBittorrent search-engine #121