let4be / crusty

Broad Web Crawler
GNU General Public License v3.0
83 stars 3 forks source link

Concurrency auto-tuning #37

Open let4be opened 2 years ago

let4be commented 2 years ago

Figure a way to auto-tune domain concurrency(there is a ~perfect N based on CPU and network bandwidth available) Will need some kind of graceful adaptive algo which will look at metrics(tx/rx, error rates) and determine the N