victorb
commented
5 years ago Sounds good to me.
Something I've noticed in the past is that in some cases (like 1 out of 10) when you create an instance in AWS, you get an instance that has lower network/cpu/disk performance for whatever reason. So I think the first benchmark we need is a instance-benchmark that when we first create an instance, we can make sure it's using the same baseline as previous ones.
Once we have some finalized specs, we can create an instance, measure it and figure out the tooling for running the same measurements on future instances.
In Go we need a way of running benchmarks on large data sets. The benchmarks would be running from benchmark repos on github and possibly take multiple hours to run.
I think having a set of 'benchmark' workers in jenkins with 'stable' resources and few TB disk space would work, but I'm open for suggestions