search

thejordanprice / p2pspider

Daemon that scrapes the DHT swarm and an express bootstrapped front-end connected to the same mongodb to create an automated magnet db with search.
https://github.com/thejordanprice/p2pspider
MIT License
20 stars 10 forks source link

P2PSpider config #5

Closed DarkPointer closed 6 years ago

DarkPointer commented 6 years ago

Hi again!

I was wondering about this part of the code : const p2p = P2PSpider({ nodesMaxSize: 250, maxConnections: 500, timeout: 1000 });

What are the expected results if the values of nodesMaxSize and timeout got changed, how will these two affect the speed of the spider and what are their impact on the host system?

Are the default ones optimal values? Do we have to edit the values according to the count of daemon instances ?

Thanks.

thejordanprice commented 6 years ago

Honestly you will have to tinker to find a good max for each, I will say the more connections and the longer the timeout the more resources you will be using... You may get better performance on the Pi running with a lower connection count. Let me know what you end up finding; I'm always curious to how well it is running on the Pi's.

DarkPointer commented 6 years ago

I just tried to set nodesMaxSize up to 5000 and it made the performance much slower I don't know if the higher maxnodes size will allow the spider to retrieve more magnets but bigger nodes max size caused the system to hang and made the crawling process much slower. Instead I tried to just double it to 500 but the results weren't much different from what they were with the default 250. However raising the timeout up to 5000 didn't have any side effects on the system performance and boosted the crawling with about +200 magnets/hour. I will keep testing and post the results here later.

thejordanprice commented 6 years ago

I think you may have more success attempting to lower the connection limit and run more copies of the daemon. In the very beginning of the project I did experiment a lot with those numbers; in attempt to find a happy spot for everyone's machines. Thanks for testing and I will poke around with the timeout on my end and see how things are at those levels.

thejordanprice commented 6 years ago

After some experimentation I have noticed that the RAM usage and connection count (possibly dead ends) will be higher with a raised timeout limit, but feel free to set it at whatever you see fit. I do notice that you will get boost in indexed magnets as you had explained. I am still gonna leave the default at 1000ms just for people that start using it and they can adjust accordingly. We've got this documented pretty well now, I'm going to close the issue.