Queue sharding support - Githubissues

let4be / crusty

Broad Web Crawler

GNU General Public License v3.0

83 stars 3 forks source link

Queue sharding support #4

Closed let4be closed 3 years ago

let4be commented 3 years ago

It's partially here, but need to add

routing to proper shard based on addr_key
spawn a green thread for each owned shard(shard_min .. shard_max)
glue it all together and test

let4be commented 3 years ago

This is now in master