search

google / tachometer

Statistically rigorous benchmark runner for the web
BSD 3-Clause "New" or "Revised" License
671 stars 22 forks source link

Disabling autosampling #223

Closed guybedford closed 3 years ago

guybedford commented 3 years ago

When explicitly setting the number of samples it would be useful to be able to entirely disable autosampling.

This would be an alternative to https://github.com/Polymer/tachometer/issues/221.

Autosampling breaks down when dealing with network testing - I'm simulating a server with network variations here and autosampling is continuning runs after the 1 minute timeout (the lowest possible timeout as well!).

As a result, running a large number of network tests is taking more than 4 hours, when the individual tests should be possible to run within 2 hours.

guybedford commented 3 years ago

Reading through the source, I see explicitly setting timeout: 0 can avoid this batch of 10 which is hard-coded. Note that the docs mention the default timeout is 0 which was why I did not attempt this.

aomarks commented 3 years ago

Thanks for the report. I agree this isn't very clear in the docs. I've sent out https://github.com/Polymer/tachometer/pull/224 which hopefully makes this more obvious:

aomarks commented 3 years ago

You may also be interested in this feature request to make the sampling process more interactive and forgiving: https://github.com/Polymer/tachometer/issues/71. I'll bump this up in my priorities, since I think it would improve usability a lot!

guybedford commented 3 years ago

Thanks so much for the updates here, that looks very clear to me.