tinylibs / tinybench

🔎 A simple, tiny and lightweight benchmarking library!
MIT License
1.73k stars 36 forks source link

perf: use `samplesLength` instead of multiple use `samples.length` #59

Closed Dunqing closed 11 months ago

Dunqing commented 11 months ago

If the number of samples is very large, calling sort() can be very expensive.

Aslemammad commented 11 months ago

Do you have any benchmark that approves these changes?

Dunqing commented 11 months ago

Here are the results of my benchmark. 2.x faster than before

┌─────────┬─────────────────┬─────────┬────────────────────┬──────────┬─────────┐
│ (index) │    Task Name    │ ops/sec │ Average Time (ns)  │  Margin  │ Samples │
├─────────┼─────────────────┼─────────┼────────────────────┼──────────┼─────────┤
│    0    │ 'before change' │   '0'   │ 1024660996.7470169 │ '±4.00%' │   50    │
│    1    │ 'after change'  │   '2'   │ 405250253.31020355 │ '±1.01%' │   50    │
└─────────┴─────────────────┴─────────┴────────────────────┴──────────┴─────────┘```

These codes for benchmark

import { Bench as BeforeBench } from 'oldtinybench';
import { Bench } from 'tinybench';

const bench = new Bench({
  iterations: 50,
});

const testFunction = async (BenchC: typeof Bench) => {
  const bb = new BenchC({ time: 100 });

  bb
    .add('faster task', () => {
      // console.log('I am faster');
    })
    .add('slower task', async () => {
      await new Promise((r) => setTimeout(r, 1)); // we wait 1ms :)
      // console.log('I am slower');
    })
    .todo('unimplemented bench');

  await bb.run();
};

bench
  .add('before change', () => testFunction(BeforeBench))
  .add('after change', () => testFunction(Bench));

await bench.warmup();
await bench.run();

console.table(bench.table());
Aslemammad commented 11 months ago

based on the new changes, can you send the new benchmarks? I'd appreciate it.

Dunqing commented 11 months ago

based on the new changes, can you send the new benchmarks? I'd appreciate it.

Looks like the changes have little difference in performance

➜ npx tsx  ./src/test-tinybench.ts
┌─────────┬─────────────────┬─────────┬────────────────────┬──────────┬─────────┐
│ (index) │    Task Name    │ ops/sec │ Average Time (ns)  │  Margin  │ Samples │
├─────────┼─────────────────┼─────────┼────────────────────┼──────────┼─────────┤
│    0    │ 'before change' │   '0'   │ 1023757644.1144943 │ '±3.63%' │   50    │
│    1    │ 'after change'  │   '0'   │ 1019888030.8961868 │ '±3.74%' │   50    │
└─────────┴─────────────────┴─────────┴────────────────────┴──────────┴─────────┘
Aslemammad commented 11 months ago

Let's remove the sort as I think it's better to not have it, but please also mention that in the docs. I believe the min/max approach is much better.

Dunqing commented 11 months ago

Let's remove the sort as I think it's better to not have it, but please also mention that in the docs. I believe the min/max approach is much better.

p75, p99, p995, p999 depend on sort(). I haven't figured out a way around that yet.