Closed mckinsel closed 6 years ago
Thanks @ryan-williams ! Definitely agree about factoring the merge scripts better. Given that it's kind of early days, I'm leaning a little toward duplication rather than abstraction, but that'll have to change soon.
I made a few changes to run the tests multiple times, and now the table reports some summary stats for 10 executions. I also switched from EBS to a local SSD, which as expected changes things a lot. We'll have to think about recording those kinds of things nicely, but for now it's a note in the README.
this is really cool @mckinsel, sorry for the delay here! it's good to merge afaic
two ideas for developing this longer-term that come to mind are:
merge_*.py
sextensions to the "Results" table:
I've wondered about the right way to deal with a benchmark-results artifact like this in many contexts (optimizing for liveness, seamless integration into existing workflows, collecting data from a representative sample of environments, etc.) so those are just a few ideas about that I'd be excited to pursue here