search

thejordanprice / p2pspider

Daemon that scrapes the DHT swarm and an express bootstrapped front-end connected to the same mongodb to create an automated magnet db with search.
https://github.com/thejordanprice/p2pspider
MIT License
20 stars 10 forks source link

No magnet links found #6

Closed alxhotel closed 6 years ago

alxhotel commented 6 years ago

So I've followed all the instructions in the README file, but after a while I don't get any magnet links.

There isn't any error logs, so it should be working.

I don't completely understand how DHT works, but I think I should be getting announce_peer in order to have a magnet link, and I'm not getting any of them. I think this is the problem.

Is there any other reason why this could be happening?

Edit: Operating System: Ubuntu 16.04 NodeJS: 9.2.0

alxhotel commented 6 years ago

Ah my bad. I left it running over night. It got up to 32 magnet links.

Is there a way to make it go faster (= find more magnet links)?

thejordanprice commented 6 years ago

What kind of hardware are you running it on? (os, ram, connection, ect)

Just to make sure, you gotta have a port forwarded to 6881 by default.

I will surely help you index faster! 😄

Sorry for the late reply. 😦

alxhotel commented 6 years ago

OS: Ubuntu 16.04 RAM: 8GB Connection: 30 Mbps down / 40 Mbps up NodeJS: 9.2.0

Port forwarding is enabled to 6881.

Thanks

thejordanprice commented 6 years ago

I'd say you could run it in cluster mode via the ecosystem.json, limit each instance of the daemon to 300MB and then set the amount of daemons to like 16.

IF the port is open and the code successfully is running, you can check with pm2 list, pm2 logs, pm2 monit. The speeds should be way higher.

Sorry for the late response, again.

alxhotel commented 6 years ago

Changed the number of daemons to 16...

Got 16,000 magnets through out the night (aprox. 12 hours) :)

Is that a normal speed ?

thejordanprice commented 6 years ago

Sorry for the delay, depending on the connection you are connected to that may be normal... I have had it hitting over 50,000 per hour. I hope this helps, if not open another issue and I'll try to explain more / help.