search

let4be / crusty

Broad Web Crawler
GNU General Public License v3.0
83 stars 3 forks source link

Provide example configurations for various setups #3

Closed let4be closed 2 years ago

let4be commented 3 years ago

This could better explain what the primary scaling points are, config for aws's c5.metal would be quite different from t2.micro

let4be commented 3 years ago

Partitially solved by lazy.sh and so called profiles https://github.com/let4be/crusty/tree/master/infra/profiles/c5.metal

But this needs further improvements

let4be commented 2 years ago

Better solution would be to configure everything in the same config file(to avoid duplication) and simply provide alternative values for several key parameters(scaling points, such as resolver concurrency and domain concurrency) and simply load approapriate values on config parse stage(by a switch via cmd option/env var)

might come in handy: https://github.com/mehcode/config-rs