Inquiry About Directing Multiple DHT Crawlers to a Single Database

bitmagnet-io / bitmagnet

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.

https://bitmagnet.io/

MIT License

2.38k stars 94 forks source link

Inquiry About Directing Multiple DHT Crawlers to a Single Database #95

Closed cnZary closed 8 months ago

cnZary commented 8 months ago

I am considering directing multiple DHT crawlers towards a single database. My primary concern is whether this setup might result in identical data entries being repeatedly written to the database. Additionally, I am curious if there might be other potential issues that could arise from such a configuration.

mgdigital commented 8 months ago

Hi, this should work but will obviously put more load on the database which might slow down the API and Web UI. It shouldn't cause any data issues.