search

google / tachometer

Statistically rigorous benchmark runner for the web
BSD 3-Clause "New" or "Revised" License
658 stars 21 forks source link

Allow user to pause/resume sampling #71

Open aomarks opened 5 years ago

aomarks commented 5 years ago

Right now If you want to stop sampling before the timeout, you can Ctrl-C, but that will kill the process without showing any results at all. And if you want to keep going after the results are in (e.g. because you want a tighter confidence interval), you have to start over entirely which means throwing away the samples you've already done.

Ideas:

  1. If the user sends SIGINT (Ctrl-C) at any time, then we should show the results we have so far, gracefully shut down, and exit.

  2. After the initial results come in, we should pause rather than exit, so that the user has the option to keep sampling.

  3. If the user presses [p]ause while sampling, we should show the results so far, and pause for more input.

  4. If the user presses [r]esume while paused, we should continue sampling (maybe for 60 seconds each time? maybe you can type how many minutes to go?)

  5. When you're running a non-headless browser, it might be nice to pause for a few seconds every minute or so, because if the browser is stealing focus, it can be hard/impossible to send any input to the runner terminal.

(6. This should probably be another issue for another time, but the entire UI could also be re-worked to fully control the terminal, and update a single table continuously using ANSI clear and cursor control sequences, so you'd just watch the intervals tighten up, and press [p] or quit when you want. My concern would be that it might make it tempting to mentally extrapolate a false convergence and draw a conclusion too early).

cc @justinfagnani @sorvell for comment

guybedford commented 3 years ago

Perhaps having the ability to configure auto-sampling modes could be useful depending on the test runner type (eg running interactively or not). For example maybe something like autosampling: 'interactive' | 'horizon' | 'timeout' | 'none'.