ghost
commented
5 years ago Hey, sorry for the delayed response, we are currently quite busy preparing the next major release. A pull request to improve to the documentation would definitely be most welcome, thanks! When you have been using something for years, it's sometimes so natural to you that it can be difficult to describe in simple terms :)
I'll try to answer your questions as best as I can:
Can we have a clear description of what kind of tests dune accepts and how to declare each test?
In a nutshell, dune doesn't know about tests specifically. There is just a standard "runtest" alias to which one can attach various commands that are usually tests. An alias is simply the grouping of various files and actions to execute under a single name.
So to define a test that is implemented by a standalone binary, one simply attaches the fact of executing this binary to the "runtest" alias. This can be done via the generic "(alias ..)" stanza or more conveniently via the "(test ...)" stanza which merges both the definition of the executable and attaching running this executable to the "runtest" alias. This is what is described as a "custom" test in the documentation.
In any case, dune has out of the box support for the diff/promote workflow: i.e. the expected output of the test is stored in a file and dune will compare the actual output to the expected one. In case of mismatch, dune will print the differences and offers a simple way to replace the committed expected output by the actual one. This is done by running dune promote.
In addition to all this, dune support inline testing, i.e. tests that are defined directly inside the code of a library. Because there are different inline testing frameworks out there, and because it is generally better to keep the core system agnostic of such frameworks, dune offer a generic inline test interface allowing third-party projects to define their own framework, or "backend". One must add the "(inline_tests)" field to a library to declare that the library contains inline tests, then dune will try to automatically discover what backend to use by scanning the library dependencies and used ppx rewriters. We do this because it is often the case that a backend is tied to a particular ppx rewriter or library. This is more convenient for the users as this avoids duplicating the information.
Why is declaring an inline with qtest is so different than ppx_inline?
In the case of qtest, tests are inside comments and the qtest backend is not tied to a particular ppx rewriter or library dependencies one would use at runtime, so it is necessary to specify it manually via (backend qtest.lib).
The examples only show a target to a (library) is that the only place where I can add inline tests? That information should be included in the documentation. How can we add inline tests to executable stanzas? There is no workaround in #745
It's currently not possible, you can only add inline tests to libraries. We'll probably lift this limitation in the future, however it requires a bit of work.
What are single tests? I found an explanation on how to run single tests, but no description of what a single test is. How to declare a single test? Are single tests the same as custom tests?
It was meant to describe how to run only one test rather than run a whole bunch of tests. When you do dune runtest, dune will run all the tests it can find. I'm not a native english speaker, so I might have got the sentence wrong in the manual.
Alizter
Hello everyone,
I am trying to add testing to a project, but I'm having trouble navigating dune's documentation. I did a quick review of this section and would be happy to contributed fixes.
The section on testing starts with:
The documentation starts by saying how to run tests, but does not cover how to declare tests, nor what kind of tests exist.
Continuing reading, I understand that there are single tests and inline tests (implicitly, from the sub-section and section titles). It would be nice to make this explicit in the documentation.
Inline tests, it appears, can be declared by adding a
(inline_tests)and(preprocess (pps ppx_inline_test))or with(inline_tests (backend qtest.lib)).qtestis so different thanppx_inline?(library)is that the only place where I can add inline tests? That information should be included in the documentation.