search

searxng / searxng

SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled.
https://docs.searxng.org
GNU Affero General Public License v3.0
12.62k stars 1.35k forks source link

Provide results from DHT, IPFS, and P2P networks #940

Open GreenLunar opened 2 years ago

GreenLunar commented 2 years ago

@seniorm0ment

IPFS would be cool and DHT crawling could be relevant here too

IPFS search is available at https://github.com/ipfs-search/ipfs-search

DHT crawler of BTDigg is available at https://github.com/btdigg-org/dhtcrawler2

Retrieve results also with Advanced Direct Connect (ADC), Ares, eDonkey2000 (eD2k), giFT, Gnutella, Gnutella2 (G2), Kad (Kademlia), OpenFT, OpenNap (Napster), Soulseek, Tribler, WASTE.

The priority for eD2k, Gnutella and Kad is not urgent because, as far as I know, every file sharing program that supports eD2k, Gnutella and Kad already has a built-in search engine for eD2k, Gnutella and Kad, but that would be a killer feature, non the less.

While I would not mind having engines for websites that index eD2k links, I meant to bind searx directly to eD2k (or Gnutella or whatever) file indexers that do aMule, eMule, Shareaza, Quazaa and others.

Originally from: searx/searx/issues/418

julianfairfax commented 2 years ago

Thing is, this would mean that SearXNG would have to actually crawl and make an index, rather than just making a request to an external site. This would take a lot more resources, especially in terms of storage.

On another note, maybe it's just me but I can never get BTDigg searching to work. On the instance I'm currently using, it just says too many requests and that it then has a reliability of 0.

Worse yet, as far as I can tell, BTDigg is the only open source torrent site, but since it doesn't have seed counts, I can almost never find a good torrent.

Is there a way to "fix" this? If not, are there any other open source torrent search engines? If not, are there any that are actually modern and not filled with ads?

It seems Library Genesis has managed to be a modern and ad-free site, since they seem to be treating their work as a public service rather than a commercial endeavour, but it seems this isn't the case for torrent sites?

I'd love to be proven wrong about any of this, but so far this is just what I've observed.

GreenLunar commented 2 years ago

Thing is, this would mean that SearXNG would have to actually crawl and make an index, rather than just making a request to an external site. This would take a lot more resources, especially in terms of storage.

Indeed. However, this is optional. If instance owner wants to enable it or not.

On another note, maybe it's just me but I can never get BTDigg searching to work. On the instance I'm currently using, it just says too many requests and that it then has a reliability of 0.

I know. I think they should take care of it. Did you try their onion instance?

Worse yet, as far as I can tell, BTDigg is the only open source torrent site, but since it doesn't have seed counts, I can almost never find a good torrent.

Exactly, but once they do have, I think som bad type of attention will come. We can have seed count as it is data that has to be received upob result count.

Is there a way to "fix" this? If not, are there any other open source torrent search engines? If not, are there any that are actually modern and not filled with ads?

Yes Torrent Paradise had a good API and was also available o IPFS https://torrent-paradise.ml/ (offline) It might be still available on IPFS. I need to check.

It seems Library Genesis has managed to be a modern and ad-free site, since they seem to be treating their work as a public service rather than a commercial endeavour, but it seems this isn't the case for torrent sites?

I'm not sure. Please elaborate. I know two torrent websites with free for personal use or public domain content http://frostclick.com/ https://librivox.org/

I'd love to be proven wrong about any of this, but so far this is just what I've observed.

I agree with most of what you've wrote above.

GreenLunar commented 2 years ago

Is there a way to "fix" this? If not, are there any other open source torrent search engines? If not, are there any that are actually modern and not filled with ads?

I've just found a torrent indexer that provides results in an RSS feed (clean from adds and scripts), but the number of seeds appears to be inaccurate, to say the least.

I think the best way is to download a list of hash checksums (magnet link or torrent files) from a single or multiple sources and then let a desktop software to retrieve information of peers per torrent.

You can also use Tribler https://www.tribler.org/ I think Tribler's search feature is the best of its kind.

julianfairfax commented 2 years ago

Is there a way to "fix" this? If not, are there any other open source torrent search engines? If not, are there any that are actually modern and not filled with ads?

I've just found a torrent indexer that provides results in an RSS feed (clean from adds and scripts), but the number of seeds appears to be inaccurate, to say the least.

I think the best way is to download a list of hash checksums (magnet link or torrent files) from a single or multiple sources and then let a desktop software to retrieve information of peers per torrent.

You can also use Tribler https://www.tribler.org/ I think Tribler's search feature is the best of its kind.

I tried Tribler but it seems I could just subscribe to torrent indexers, which doesn't change much. Maybe I was using it wrong?

GreenLunar commented 2 years ago

I don't think you can go wrong, so you definitely used it's built-in search. Perhaps Tribler needs more sharers.

I've found a few torrents, though very slow ones due to rarity, that are not to be found in any torrent indexer. Currently, I've over 500 indexers in my bookmarks (not all of them online and some are of the same brand).

Nevertheless, rest assured that you are safe from ads with Tribler ;)