Cobench user library

[art-w]

The expected "json output" on make bench can be a blocker for new users, since it's not documented enough. It also limits the schema changes we can do to add new functionalities.

It would be nice to have a friendly library like Alcotest to ease the mental cost of creating benchmarks.

1. [ ] The first version can be really simple: Just a function to output a metric in the expected format!

   val metric : name:string
            -> ?units:[`seconds | ...]
            -> ?trend:[`lower_is_better | ...]
            -> ?descr:string
            -> ...
            -> float
            -> unit

2. [ ] Document how to setup a dune rule @bench and a Makefile bench rule?

3. [ ] Would be nice to have a test_case interface, where the elapsed time and other metrics are recorded for the user. The target here is the "common case" benchmarks for libraries that just record the raw performance of their operations (ie have a default setup that provides a lot value, with timings, GC info, etc.)

4. [x] Add an alternative way to push the metrics with the new API

We jokingly talked about this as Alcobench, but unless it provides a really colored output, it's not fair to steal their name! Perhaps Cobench for Continuous Benchmarking would fit better? Any other ideas?

[shubhi-arora]

Hi @art-w , I am an outreachy applicant, i want to work on this issue, but need guidance on how to start with it, can you please help me out?

[art-w]

Hello @shubhi-arora, thanks for taking a look at this!

We have a very rough draft of the code for this issue in the cobench folder of this repository, with a lot of things missing! A good start would be to add a new function val print : t -> unit that outputs its argument in json format (with the existing to_json + Yojson.Safe.to_string + print_endline)

If you are getting started with OCaml, you'll need to first install opam in order to then install dune (to build and run the code) and the json library yojson. See also these tutorials for getting started :)

Feel free to ask as many questions as you need!

[gs0510]

I'm going to take number 1 on this issue :)

[ElectreAAS]

@art-w do you think this is still necessary after #422 and Cb-check? I'm asking because cobench is the only reason we have a dependency on mtime, which proves to be a little annoying

[art-w]

In its current form, I don't think anyone uses cobench -- but I do think it would be easier for projects to use a library than to learn the json format? (cb-check answers the problem of validation, not so much of producing?)