Check existing data when starting a crawl to not mix data from different networks

dennis-tra / nebula

🌌 A network agnostic DHT crawler, monitor, and measurement tool that exposes timely information about DHT networks.

Apache License 2.0

294 stars 30 forks source link

Check existing data when starting a crawl to not mix data from different networks #29

Open dennis-tra opened 1 year ago

dennis-tra commented 1 year ago

Imagine you're running the crawler for the IPFS network for some time. Then you want to start crawling the FILECOIN network as well and experiment around. This could easily lead to FILECOIN data ending up in the same database as the IPFS data. This could be avoided if prior to each crawl we check which network was actually crawled before.