search

ltratt / multitime

Time command execution over multiple executions
http://tratt.net/laurie/src/multitime/
MIT License
115 stars 13 forks source link

Feature: Confidence Interval #29

Open Lewiscowles1986 opened 2 years ago

Lewiscowles1986 commented 2 years ago

13 and #28 deal with the confidence level. One is a query, and the other is a short-term fix.

This issue seems like a good place perhaps @snim2 @ltratt and I (anyone interested, I've invited myself 😄); could document the vision, progress and usage of the confidence score before documenting it on the README?

ltratt commented 2 years ago

I must admit that it has been some considerable time since I've done any work on multitime, so my memory is pretty fuzzy. That's partly because multitime does most of the basic things I need -- though I would really like to it to output confidence intervals by default! It's also partly because I haven't been sure if an "advanced" version of multitime would be useful given the existence of tools like https://github.com/sharkdp/hyperfine (which I have only very very briefly tried). My gut feeling is that multitime might still occupy a useful niche because it's simple, lightweight (and is a drop-in replacement for the normal unix time command).

In terms of confidence intervals, my vague thought is that a good approach might be the following:

  1. When called as unix time, output text in traditional time format (for those tools that rely on it). This is the current behaviour, so no change is needed.
  2. If called with -n, try using confidence intervals when possible. I would need to check the maths again, but from memory one can generate better quality confidence intervals when -n is greater than 30. My guess would be that when -n is small (at the limit: 2) it might be better to use something like multitime's current output format (which doesn't do much data processing at all) to avoid misleading the user. But if -n is "big enough" I would probably default to outputting confidence intervals a succinct format such as md5 -t: real 0.037s +/- 0.002s (95% confidence interval with a more verbose mode giving mins and maxes etc.
  3. If the user doesn't specify a confidence interval, I'm not sure if we should supply a default (research papers tend to use 95%, but I have preferred 99% in my own work), and whether that default should be fixed ("always 95%") or perhaps increase as -n increase (so e.g. -n==10 95% but -n==100 99%).

Warning: I might not have thought about these things sufficiently!

snim2 commented 2 years ago

Hello both!

I have finally managed to dig out an ancient version of multitime which contains my confidence interval code. Perhaps a reasonable place to start would be for me to make a PR with that version of the code, maybe pull it up to date if it's a long way behind master and perhaps we can figure out why we didn't think it was ready to merge in the first place!

ltratt commented 2 years ago

If you've got time to do that, I'm certainly willing to review and offer comments at a minimum!