search

isohuntto / openbay

Other
3.53k stars 903 forks source link

Automatic decentralised update function using the dht #157

Open chinwendu opened 9 years ago

chinwendu commented 9 years ago

https://github.com/chinwendu/openbay-crawler

I finally released the first 100% independent decentralised automatic update function for the openbay projet. I do not see it as a replacement for an RSS update, but more like a complement. Once you launch it, it will listen to the dht for torrents requests. On torrent request (announce or peer request), it will download the torrent file from the peers found in the dht or try to download it from torcache if none of the peers have responded. It will then feed the database based on the metadata inside the torrent file (name, list of files, size, creation date ...).

I made a function to distribute the info_hash and torrent url to others running the program to distribute the crawling effort of the dht between the participant.

I personally feed my own database with almost 3.000.000 torrents (starting from scratch) in less than a month.

I love to hear some feedback from openbay users, so feel free to comment.

edit: I forgot to say it can also scrape torrents againts udp://open.demonii.com:1337 and update to db with seeder and leecher number.

TPBT-OFFICIAL commented 9 years ago

Unfortunatly i can't seem to make it work i keep getting some weird python errors if you could help me i would be glad to show you what kind of errors i get.

chinwendu commented 9 years ago

sure no problem, if you can give me a traceback

edit: and the OS you are using

TPBT-OFFICIAL commented 9 years ago

OS: CentOS 6 Errors: When i run "./crawler.py" i get Traceback (most recent call last): File "./crawler.py", line 14, in from btdht import DHT, ID, RoutingTable File "/usr/lib64/python2.6/site-packages/btdht/init.py", line 13, in from .dht import DHT, DHT_BASE, ID, Node, Bucket, RoutingTable, NotFound, BucketFull, NoTokenError File "btdht/krcp.pxd", line 3, in init btdht.dht (btdht/dht.c:48774) File "btdht/krcp.pyx", line 19, in init btdht.krcp (btdht/krcp.c:19498) File "btdht/utils.pyx", line 15, in init btdht.utils (btdht/utils.c:12432) ImportError: cannot import name total_ordering

chinwendu commented 9 years ago

I think you need python 2.7 https://docs.python.org/2/library/functools.html#functools.total_ordering New in version 2.7

edit: I added it as a dependacy in README.md, did think to do it as python 2.7 is the version distributed with the last debian stable (wheezy)