Corebench Roadmap

[deckarep]

Corebench Roadmap 2018

Here's the roadmap for this tool, if anyone wants to take a crack at one of these please do the following:

Contributing

1. Create a new issue with the title: Feature: xyz
2. Ping @deckarep in the ticket with your intent to contribute
3. Ask clarifying questions as needed around design approach
4. Work on ticket, add your name to AUTHORS.md file
5. Ensure build passes and ping @deckarep when finished for a merge

Immediate Needs

• [x] Implement a nice logger package, make it pretty with colors 🤗
• [x] Add the ability to choose instance sizes based on CPU count
• [ ] Integrate Benchstat command as a flag, for doing consecutive runs to eliminate noise and delta compares using --stat flag.
• [ ] Cleanup/speedup polling logic and reduce time to bench [ provisioning -> benchmarking ]
• [ ] Allow the ability to use either ENV vars OR FLAGS...to make CLI easier.
• [ ] Ability to go get all dependencies if they aren't vendored in using /vendor
• [ ] Ability to specify Go versions as command-line flag
• [ ] Cleanup: go vet, linting, etc
• [ ] Implement terminal spinner so user's know work is happening while waiting.
• [x] Support regex for benchmarks to specify certain ones to run
• [ ] Ability to tee benchmark data to a local file for later examination
• [ ] Ability to capture: cpuprofile, blockprofile, memprofile, etc to local file for later examination
• [ ] Ability to capture binary for usage with pprof tool
• [ ] On panic, ensure proper termination of a cloud resources
• [ ] Add at least another provider: Google Cloud (since they charge by the minute)
• [ ] Add a --dryrun command, which indicates what corebench will do.
• [ ] Stabilize the API (at least add one more provider)
• [ ] Unit tests
• [ ] Better SSH key handling and installation (so you can log in and inspect box)

Future Needs

• [ ] Ability to generate comprehensive graph reports from benchmark data
• [ ] Support default .corebench configuration of: region, droplet size, basic settings
• [ ] Add more providers: AWS, ???
• [ ] Ability to benchmark non-open source code via rsync
• [ ] Ability to launch your own profiling container with your own OS, tooling, etc.

Way, way future

• [ ] Ability to kick off parallel benchmarks...what's the use-case here?
• [ ] Integration with Slack upon completion, or Email report when finished (fire and forget)

[deckarep]

@chewxy - do you think you'll be able to contribute the AWS provider? If so, i'll probably start looking at implementing the Google provider.

[chewxy]

actually my colleague @jokebroker will be picking up the AWS provider. I'll be providing codereview support (rest assured he can write good Go code)

[chewxy]

YSK that benchmarking on cloud systems are not suitable for microbenchmarks btw. There is a LOT of noise for anything that is sub microseconds. The main utility of corebench is benchmarking larger algorithms (like neural networks).

[deckarep]

@chewxy - glad to hear that...yes I am aware of cloud systems due to things like noisy neighbor and noise on top of the virtualization layer.

I don't consider the microsecond resolution to be accurate but it's still nonetheless useful to gauge vertical scaling of applications across cores which is my current target.

Thanks!

[deckarep]

Also, looking forward to @jokebroker's contribution! :)

[virtuallynathan]

FWIW, AWS has per-second billing: https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/

[deckarep]

@virtuallynathan - yes i need to update the docs I think both AWS and Google are now on per-second billing. Both are nice in terms of saving 💰 👍