Please allow optional setUp and tearDown methods

[emchristiansen]

I'd like to run a benchmark that requires initializing a threadpool before the benchmark and closing it after the benchmark. Currently, it seems to be impossible to exclude the initialization / closing overhead from the timings collected by Benny.

Could you please enable this use case? You could do it by accepting optional "setUp" and "tearDown" functions in the add method.

[vjpr]

I need this too.

Not sure what the API should look like - lots of options.

I love the convenience of returning a test function which makes use of the setup scope.

To note: the author prefers functions over chainable methods as mentioned here. I think its a pretty good idea too.

I wrote a benchmarking framework recently and went down the chainable approach ava-style.

Options

// Return array for test and cleanup - makes use of closure for convenience

b.add('foo', async () => {
  // setup
  return [
    async () => {
      // run test
    },
    async () => {
      // cleanup
    },
  ]
})

// Returns object for test/cleanup - convenient closure

b.add('foo', async () => {
  let pool
  return {
    test: async () => {
      // run test
    },
    setup: async () => {
      // setup
      pool = new Pool
    },
    cleanup: async () => {
      // cleanup
      pool.destroy()
    },
  }
})

// Fluent - pass context/builder object like ava

b.add('foo', async t => {
  t.setup(async () => {
    t.context.pool = new Pool()
  })
  t.cleanup(async () => {
    await t.context.pool.destroy()
  })
  t.run(async t => {
    t.context
  })
})

// Fluent - test func as first arg

b.add('foo', async t => {
  // setup
  return async () => {
    // test
  }
})
  .cleanup(async t => {})
  //.setup(async t => {})

// Fluent - allow methods as params

b.add(
  'foo',
  async t => {
    // test
  },
  async () => {
    // setup
  },
  async () => {
    // cleanup
  },
)

[CMCDragonkai]

It seems that tests can have a setup before the benchmark takes place. But I wonder if it is possible to do it all before the suite is started, then to shut it all down in the complete callback.

[geelen]

I've been using top-level await and https://github.com/esbuild-kit/tsx to run my benchmarks, works ok!

import b from 'benny'

const {app, queue} = await someGlobalSetup()

await b.suite(...)

await b.suite(...)

await b.suite(...)

await queue.onIdle()
await app.close()

[CMCDragonkai]

@geelen so in that case setup and teardown happen before the whole suite and after the whole suite.

But there's no setup and teardown per added bench as per @vjpr.

Since this repo is a bit dead, maybe someone can fork it and add this feature?