semgrep / testo

Test framework for OCaml
ISC License
16 stars 1 forks source link

Provide a way to custom-test the contents of captured output #18

Closed mjambon closed 8 months ago

mjambon commented 8 months ago

The current mechanism for checking stdout or stderr compares the output of the test with previously captured output stored in the snapshots folder. This is fine but sometimes the output is variable due to timestamps or temporary file paths that differ from one run to another. This problem is mostly solved by masking the variable parts of the output before the comparison. However, this gets impractical when we're only looking for a specific substring or pattern in the output and we don't care about the rest. To address this problem, I suggest adding an option to customize the comparison of captured output:

type check_output = Diff | Custom of (old:string -> new_:string -> unit)

This would be an extra option of the Testo.create function. Diff would be the default, built-in way to compare captured output. Creating a test would look like this:

let test_foo =
  Testo.create
    ~checked_output:Stderr
    ~check_output:(Custom (fun _old new_ -> contains new_ "some important message"))
    "foo" (fun () -> ...)
mjambon commented 8 months ago

This was already mostly supported in the form of output masking as long as the output can be converted into a string to be compared with a reference.

The masking/normalization method is not applicable for comparing e.g. a float against an other float within a margin of error such as fun old new_ -> abs_float (new_ -. old) < 1..

mjambon commented 8 months ago

The remark above makes me wonder if we should rename mask_output with normalize_output which reflects its generic nature better.