search

kevinlynx / dhtcrawler2

dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
Other
460 stars 658 forks source link

Not finding any torrents #2

Closed Ivshti closed 11 years ago

Ivshti commented 11 years ago

Hello,

Are there any special network requirements in order to get the DHT crawler running?

I opened the 6776 (UDP and TCP) port on my router, but still - after following the instructions on the README (starting crawler/hash/http) and waiting 10 minutes - the crawler has not found anything yet.

Stats:

total 0 torrents

2013-07-10 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0
2013-07-09 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0
2013-07-08 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0

Any ideas?

kevinlynx commented 11 years ago

crawler need UDP ports ranged 6776 -> 6776+node_count, received 0 queries means the crawler can't receive messages from other dht nodes. Or the crawler join dht network failed. Crawler will first join dht.transmissionbt.com node, make sure it can resolve this domain ip address.

Ivshti commented 11 years ago

I pinged dht.transmissionbt.com (it's OK) - also I disabled iptables on a machine with a public IP and I manually checked those ports - they're open.

Is there a way to enable more detailed logging so I can get an idea of what happens? The only thing I know for sure is that, when I start the crawler, absolutely no network traffic is used.

Is it possible that the problem is that I'm running the build on Linux? Or maybe some configuration?

kevinlynx commented 11 years ago

see priv/dhtcrawler.config, set loglevel to 0 to see trace logs

Ivshti commented 11 years ago

Nevermind, the problem was that I was starting the crawler like that: "sh win_start_crawler.bat &" and it was apparently killed after 1-2 minutes of running.