search

boramalper / magnetico

Autonomous (self-hosted) BitTorrent DHT search engine suite.
http://labs.boramalper.org/magnetico/
GNU Affero General Public License v3.0
3.05k stars 341 forks source link

Add support for crawling from tracker for initial #201

Open lopoki opened 5 years ago

lopoki commented 5 years ago

Here is list for main/biggest tracker https://zooqle.com/api/json_trackers.php?limit=200

Please add support to crawl from trackers above

ad-m commented 5 years ago

Why do you think this should be a magnetico functionality, not a software that uses data collected by magnetico eg. front-end UI?

kescherCode commented 5 years ago

@ad-m I think they want magneticod to store the metadata from a tracker, effectively making your own copy of the metadata and putting it in magneticod's database.

boramalper commented 5 years ago
  1. I am not even sure how trackers can be crawled (i.e. make them dump all their torrents). You might want to share that, but...
  2. The premise of magnetico is to decentralise BitTorrent. It's not merely a to dump all the torrents out there. I am not dismissing the idea entirely, as I think becoming a Google-grade torrent search engine is an ambitious & exciting goal, but I am not fully sold either.

Anyway, if you can answer (1.), I'd be happy to consider it at least, or to write another tool to populate the database.

boramalper commented 5 years ago

Also see #197.

skobkin commented 4 years ago

I don't think that tracker import/search is what magnetico intended to do.

In my humble opinion key word in Bora's answer is "decentralise". When you're relying on trackers you're making it more centralized and less tolerable to internet censorship.

Also there are many problems with data retrieval because there is no "common" format which all trackers are using. Yes, you can use *.torrent files but not every tracker is serving them directly without registration or some sort of protection. So if you're going to implement import from torrent trackers you'll need to write data source adapters almost for each particular tracker. It couldn't be implemented in such manner when you're just giving magneticod ANY torrent tracker URL and it'd just work.

Not to mention that just maintaining and supporting such tracker list you must have a lot of time to adapt your code every time something changes.

For now magnetico is something that follows Unix philosophy's "DOTADIW" principle. And it's good because it can be simply integrated with/in other software. Making it all-in-one solution is probably not such a good idea.

ProphetZarquon commented 4 years ago

While I do actually like the idea of magnetico becoming the basis of a unified framework for all things torrent related, I agree that scraping every magnet from a tracker seems like something one might run prior to magnetico rather than something magnetico* would do.