ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.24k stars 353 forks source link

more flexible flags for advanced testing #3166

Open avsm opened 6 years ago

avsm commented 6 years ago

in opam2, we have with-test to run basic tests, but we are now running into some limitations that I wanted to discuss before the opam2 RC (in the interest of minimising opam format changes)

Any thoughts on how to incorporate these sorts of things into the current with-test and with-doc mechanism? Something slightly more general would let us expose fuzz testing and benchmarking more easily.

cc @yomimono @stedolan @gemmag who were in the discussion about wanting this feature.

dbuenzli commented 6 years ago

in opam2, we have with-test to run basic tests

with-test is not to run test, it is to indicate to build the tests.

running benchmarks is also highly desirable for some packages, and this currently has no place in the test/doc flags.

I once opened this https://github.com/ocaml/opam/issues/2278, but eventually closed as I'm quite sure it's not such a good idea (see message there).

yomimono commented 6 years ago

I'd like not to conflate the (valid, as far as I can tell) reasons for not doing this for benchmarking, with whether it's reasonable for the general case of testlike things that take a long time to run.