vitest-dev / vitest

Next generation testing framework powered by Vite.
https://vitest.dev
MIT License
12.78k stars 1.15k forks source link

Support it() as well for bench #5766

Open lveillard opened 4 months ago

lveillard commented 4 months ago

Clear and concise description of the problem

So we have already 150 tests we want to benchmark. We need to manually copy them, replace the it() by a bench() and then run our benchmarking

Suggested solution

While not optimal (we are including the expect... logic in the benchmark) it would be really convinient to be able to vitest bench the same files as vitest run

Alternative

An alternative is to store the body of the tests in an array, and then have a creator that encapsulates them as tests or as bench. But too messy.

Another alternative would be to build a different set of tests, but we want to leverage the existing ones.

Additional context

No response

Validations

hi-ogawa commented 4 months ago

It sounds reasonable, but probably for the better argument, can you provide some concrete examples of some tests you want to bench?

Without considering actual design or implementation, one concern I might have is expect(...).toEqual(...) etc... assertions has pretty non trivial code to traverse you objects, so having all the assertions inside bench can skew the result depending on the assertion.

lveillard commented 4 months ago

We have these tests for instance: https://github.com/Blitzapps/blitz-orm/blob/main/tests/unit/queries/query.ts

Is an ORM for typeDB/surrealDB. Same way we run the tests for both DBs, we would love to just change the command and just run them as bench instead of tests.

I see two alternatives:

Both would be great for us. Our next step would be finding a way to run the tests once per DB to compare them. But lets keep that one for later 😂