Support for expect tests

[samoht]

We should check that one can run an expect test, and have corrections work for that test, and then include the results in the mdx. Mostly I think this would be about getting the dune rules to work properly, rather than mdx as a tool.

(e.g. we should at least document how use ppx_expect with mdx)

[clecat]

Are the ppx_expect tests usable in the top_level ? I would be glad is someone using regularly could point me a way to do it

[clecat]

I proposed this issue, which, if accepted and implmented, would allow ppx_expect to be used in a top_level.

[clecat]

Coming back with a question: Are you sure that you want to run the ppx_expect tests inside of the markdown, or do you only want to write some ppx_expect use cases, and continue without taking care of the test result ? The thing is, mdx uses the top_level, and ppx_expect is not compatible with the top_level.

• One possible solution, which should work on short-term, would be to deactivate the test if it is run from a top_level instead of raising an error (because inline tests were not activated).
• On the other hand, we could change the ppx_expect code so it would run the actual tests as a diff when run from a top_level, but it may take more time to be done (it's far from what ppx_expect currently does, but it's still doable).

What is the behaviour you want ?

[clecat]

Third solution, might be the best (was actually the one I had in mind, but looks clear to me now):

• Instead of supporting the promotion, we do what I proposed in the issue mentioned earlier: We will not be able to give the promoted source code, but we would still have the diff mentioned.

Could be a good in between

[samoht]

Could you give an example of the workflows you are describing? I am not sure to follow.

[clecat]

Let's consider the following examples: ex 1: Correct test

# let%expect_test _ =
     let f output =
       print_string output;
       [%expect {| hello world |}]
     in
     f "hello world"

ex2: Bad test: "error" <> "hello world"

# let%expect_test _ =
     let f output =
       print_string output;
       [%expect {| hello world |}]
     in
     f "error"

Solution 1: Lazy solution

Tests activated from the top_level are ignored: The examples 1 and 2 are ignored, raising no error, but we do not know that the ex 2 if wrong

Solution 2: Complex solution

Tests activated from the top_level are run and the source code if corrected: The examples 1 and 2 are properly run, raising an error for the second case, giving us the corrected code:

Exception: Expected "hello world", got "error" instead: here is the promoted code:
  let%expect_test _ =
    let f output =
      print_string output;
      [%expect {| error |}]
    in
    f "error"

Solution 3: Intermediate solution

Tests activated from the top_level are run but no promotion is done: The examples 1 and 2 are properly run, raising an error for the second case:

Exception: Expected "hello world", got "error" instead.

The third solution looks like the best to me, but my major issue is that it may not be what you want at all: If you only want to write the tests as examples, ignoring them would be the solution as launching them implies a lot of things (ppx_expect uses code source files etc which we do not have in mdx as we run from the top_level)

[yminsky]

I wasn't thinking that expect tests would work in the toplevel at all. All I really want is to be able to write code in ordinary .ml files containing tests, and have the expect-test workflow work there in the ordinary way (detecting differences between the current state of the file and what you get by running the test, and having the opportunity to promote the generated file), and at the same time, having a workflow that can then promote changes made in those files into the mdx file proper. promote-style workflows in the toplevel itself seem hard to think about, and it doesn't really strike me as worth supporting.

[clecat]

Okay, sorry for the delay. I'm working on that, the main part is already implemented, only small changes are needed, I'll come back soon with a solution, so you can tell me if it's what you wanted

[clecat]

So, tell me if this is what you want @yminsky : Let's consider that you have the following file tree:

├── dune-project
└── src
    ├── dune
    ├── test.md
    └── test.ml

The dune file is simple:

(library
 (name test)
 (inline_tests)
 (preprocess (pps ppx_expect)))

The content of test.ml is the following:

[@@@part "1"]
let%expect_test _ =
    let f output =
      print_endline output
    in
    f "hello world";
    [%expect{| hello world |} ]

And test.md:

Mdx can also understand ocaml code blocks:

```ocaml file=test.ml,part=1
let%expect_test _ =
    let f output =
      print_endline output
    in
    f "hello world";
    [%expect{| hello world |} ]
`` `

Both files are in a good shape, having the same and correct content. Let's now imagine that you change test.ml for:

[@@@part "1"]
let%expect_test _ =
    let f output =
      print_endline output
    in
    f "hello OCaml world";
    [%expect{| hello world |} ]

We now have 2 problems: 1) The test is not correct anymore 2) Neither is the markdown file.

In order to do the promotion, I added few rules to the dune file

(library
 (name test)
 (inline_tests)
 (preprocess (pps ppx_expect)))

(rule
 (targets test.md.corrected)
 (deps (alias runtest))
 (action
  (run ocaml-mdx test --direction=to-md --force-output %{dep:test.md})))

(alias
 (name mdx)
 (action
  (diff test.md test.md.corrected)))

And here is what you do:

$ dune build @mdx
Done: 63/68 (jobs: 1)File "src/test.ml", line 1, characters 0-0:
        diff (internal) (exit 1)
(cd _build/default && /usr/bin/diff -u src/test.ml src/test.ml.corrected)
--- src/test.ml 2019-04-01 12:16:56.083225711 +0200
+++ src/test.ml.corrected   2019-04-01 12:16:56.861230604 +0200
@@ -4,4 +4,4 @@
       print_endline output
     in
     f "hello OCaml world";
-    [%expect{| hello world |} ]
+    [%expect{| hello OCaml world |} ]

The tests are run, you can promote if that's what you want

$ dune promote
Promoting _build/default/src/test.ml.corrected to src/test.ml.

Then you rerun the command

$ dune build @mdx
Done: 65/68 (jobs: 1)File "src/test.md", line 1, characters 0-0:
        diff (internal) (exit 1)
(cd _build/default && /usr/bin/diff -u src/test.md src/test.md.corrected)
--- src/test.md 2019-04-01 12:16:56.018225303 +0200
+++ src/test.md.corrected   2019-04-01 12:17:03.779274106 +0200
@@ -5,6 +5,6 @@
     let f output =
       print_endline output
     in
-    f "hello world";
-    [%expect{| hello world |} ]
+    f "hello OCaml world";
+    [%expect{| hello OCaml world |} ]
 `` `

The diff between the newly updated ml file and the md file is done. You now only have to do the promotion again:

$ dune promote
Promoting _build/default/src/test.md.corrected to src/test.md.

And that's it, you know have the following test.md file:

Mdx can also understand ocaml code blocks:

```ocaml file=test.ml,part=1
let%expect_test _ =
    let f output =
      print_endline output
    in
    f "hello OCaml world";
    [%expect{| hello OCaml world |} ]
`` `

Tell me if that's what I expected, and I will push the modifications :)

[yminsky]

That looks like a reasonable workflow. I think the two levels of promotion is a little odd (if you're willing to do one promotion, you're very likely willing to do the other), but it doesn't seem worth trying to come up with a one round solution.

y

On Mon, Apr 1, 2019, 6:20 AM Charles-Edouard Lecat notifications@github.com wrote:

So, tell me if this is what you want @yminsky https://github.com/yminsky : Let's consider that you have the following file tree:

├── dune-project

└── src

├── dune

├── test.md

└── test.ml

The dune file is simple:

(library

(name test)

(inline_tests)

(preprocess (pps ppx_expect)))

The content of test.ml is the following:

[@@@part "1"] let%expecttest =

let f output =

  print_endline output

in

f "hello world";

[%expect{| hello world |} ]

And test.md:

Mdx can also understand ocaml code blocks:

let%expect_test _ =

    let f output =

      print_endline output

    in

    f "hello world";

    [%expect{| hello world |} ]

`` `

Both files are in a good shape, having the same and correct content.
Let's now imagine that you change test.ml for:

[@@@part "1"]
let%expect_test _ =

    let f output =

      print_endline output

    in

    f "hello OCaml world";

    [%expect{| hello world |} ]

We now have 2 problems: 1) The test is not correct anymore 2) Neither is
the markdown file.

In order to do the promotion, I added few rules to the dune file

(library

 (name test)

 (inline_tests)

 (preprocess (pps ppx_expect)))

(rule

 (targets test.md.corrected)

 (deps (alias runtest))

 (action

  (run ocaml-mdx test --direction=to-md --force-output %{dep:test.md})))

(alias

 (name mdx)

 (action

  (diff test.md test.md.corrected)))

And here is what you do:

$ dune build @mdx

Done: 63/68 (jobs: 1)File "src/test.ml", line 1, characters 0-0:

        diff (internal) (exit 1)

(cd _build/default && /usr/bin/diff -u src/test.ml src/test.ml.corrected)

--- src/test.ml   2019-04-01 12:16:56.083225711 +0200

+++ src/test.ml.corrected 2019-04-01 12:16:56.861230604 +0200

@@ -4,4 +4,4 @@

       print_endline output

     in

     f "hello OCaml world";

-    [%expect{| hello world |} ]

+    [%expect{| hello OCaml world |} ]

The tests are run, you can promote if that's what you want

$ dune promote

Promoting _build/default/src/test.ml.corrected to src/test.ml.

Then you rerun the command

$ dune build @mdx

Done: 65/68 (jobs: 1)File "src/test.md", line 1, characters 0-0:

        diff (internal) (exit 1)

(cd _build/default && /usr/bin/diff -u src/test.md src/test.md.corrected)

--- src/test.md   2019-04-01 12:16:56.018225303 +0200

+++ src/test.md.corrected 2019-04-01 12:17:03.779274106 +0200

@@ -5,6 +5,6 @@

     let f output =

       print_endline output

     in

-    f "hello world";

-    [%expect{| hello world |} ]

+    f "hello OCaml world";

+    [%expect{| hello OCaml world |} ]

 `` `

The diff between the newly updated ml file and the md file is done. You
now only have to do the promotion again:

$ dune promote

Promoting _build/default/src/test.md.corrected to src/test.md.

And that's it, you know have the following test.md file:

Mdx can also understand ocaml code blocks:

```ocaml file=test.ml,part=1

let%expect_test _ =

    let f output =

      print_endline output

    in

    f "hello OCaml world";

    [%expect{| hello OCaml world |} ]

`` `

Tell me if that's what I expected, and I will push the modifications :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/realworldocaml/mdx/issues/112#issuecomment-478522426>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AArqJrkiwjot81XwjsNF8LJRPiVZcg9cks5vcd19gaJpZM4bWVGG>
.

[clecat]

Okay nice For the promotion, it seems that --auto-promote only applies at the end of the whole rule, so you will need to promote again at the end, but it will spare you 1 call to the promotion (if you are sure about auto promoting)

[clecat]

A small update: You have no need to use force-output if you use the following dune file:

(library
 (name test)
 (inline_tests)
 (preprocess (pps ppx_expect)))

(alias
 (name mdx)
 (deps (:x test.md) (alias runtest))
 (action
  (progn
    (run ocaml-mdx test --direction=to-md %{x})
    (diff? %{x} %{x}.corrected))))

(you still need to do the double promotion)