Running unit tests

[m-kru]

Are there any plans to add command for running unit tests? alr test exists, but it Tests the compilation of all or some releases.

[mosteo]

Not right now, because alr test is intended to verify integration into alire, not integrity of the release itself. But I see how it could be a useful part of the development workflow, so thanks for the suggestion.

[kalkin]

I also would like to specify some dependencies as test only, so the consumer of the library does not need e.g. aunit.

[onox]

I also would like to specify some dependencies as test only

Many package managers allow you to specify a list of "developer dependencies". These are dependencies that are not downloaded and built when the crate is used as a dependency of another crate, but only when performing certain tasks in the project of the crate. These include tools like linters, code formatters, dependencies for tests, etc.

In this case aunit should indeed be added to this list of developer dependencies. In my own projects I had to workaround this missing feature by creating an extra unpublished "tests" crate with aunit as a regular dependency.

IMO alr test should run (unit) tests and perhaps another command like alr prove could run the gnatprove tool.

[kalkin]

@onox Thanks for your reply. I solved it actually the same way you proposed :-).

[mosteo]

@onox , IIUC, this could go hand in hand with customizable user tasks? Something like:

[[task]]
name = "prove"
command = ... # what has to be run for this task
[task.depends-on]
# Extra dependencies
[task.environment]
# Extra environment

[[task]]
name = "unit-testing"
command = ... 

and then you issue alr task prove or alr task unit-testing, etc? Overkill?

[onox]

Customizable user tasks are definitely useful, but I'm not sure about specifying the dependencies per task :thinking: Other package managers like poetry, cargo, yarn/npm seem to allow you to specify just 1 list called dev-dependencies. It could work exactly like [[depends-on]], supporting 'case(os)', etc.

Would it make sense to have something like [dev-depends-on.'case(task).A|B'] so you can specify a particular dev-dependency for multiple tasks? A project may have tasks like lint:check and lint:fix, both requiring some tool.

[mosteo]

Thanks for the feedback, onox. Just exploring ideas for now. My concern is that some task my have a "scarce" dependency that might raise the entry cost for other would-be developers.

[kalkin]

@mosteo IMHO a complicated [dev-depends-on.'case… syntax may scare developers away. :see_no_evil:

[onox]

@mosteo IMHO a complicated [dev-depends-on.'case… syntax may scare developers away. see_no_evil

I meant this as a possibility for the developer to be more specific. If the developer doesn't mind to put everything in one list, then just [[dev-depends-on]] can be used.

[Fabien-Chouteau]

I am a bit scared of the complexity that will bring customizable user tasks. For the moment I am more in favor of having a command to run programs in the Alire context (e.g. $ alr exec "gnatprove ...").

That being said, SPARK might deserve its own $ alr prove command.

For [[dev-depends-on]], I think it makes sense. How Alire will know whether to get the dev depends or not?

For alr test, I think what we discussed once @mosteo is that this command would be more for a "sanity-check" of the crate, more than a full testing that may require a complex environment. After the discussion here: https://github.com/alire-project/alire/issues/721, I think the way I would like to test my crates in the future would be to have a local testing crate in a sub folder of my repo. In that create add a dependency to the main crate using a relative path, and then dependencies for testing (AUnit for instance).

[mosteo]

For [[dev-depends-on]], I think it makes sense. How Alire will know whether to get the dev depends or not?

If there are new user-task oriented commands, be it exec, task, these would require the extra dependencies and use the corresponding environment.

If we eventually do user-defined tasks, one possibility is that alr build, alr prove, alr test can be defaults in absence of corresponding user-defined tasks (that is, alr prove = alr task prove when defined, for example).

For alr test, I think what we discussed once @mosteo is that this command would be more for a "sanity-check" of the crate, more than a full testing that may require a complex environment.

This is correct. I've thought sometimes that the current alr test, which is currently kind of an internal command, might be moved under the alr dev reserved command, and leave a future alr test for proper testing of the crate according to user instructions. Although the solution proposed by Fabien is straightforward and already works.

[onox]

This is correct. I've thought sometimes that the current alr test, which is currently kind of an internal command, might be moved under the alr dev reserved command, and leave a future alr test for proper testing of the crate according to user instructions.

This makes sense to me. Let all these internal Alire commands that are of no interest to the end user be hidden behind a reserved command like alr dev.

I am a bit scared of the complexity that will bring customizable user tasks. For the moment I am more in favor of having a command to run programs in the Alire context (e.g. $ alr exec "gnatprove ...").

This basically requires providing precise instructions in a README.md or the use of a Makefile. Shouldn't proving (or testing) be easy and done often? Running alr prove or alr task prove sounds easier to me :slightly_smiling_face:

[bmilanov]

I believe that having an "alr prove" command would be nice. First, I can't think of any clashing functionality. Second, this can be used to advertise a tool that might benefit projects that are not committed to 100% proof. Is it possible for gnatprove to bring value to non-SPARK projects with some of its more lightweight checks?

[Fabien-Chouteau]

For the moment I am more in favor of having a command to run programs in the Alire context (e.g. $ alr exec "gnatprove ...").

Which can be even more useful with an alias feature a la Cargo: https://doc.rust-lang.org/cargo/reference/config.html#alias and some default aliases, e.g.:

[alias]
prove = ["exec", "gnatprove", "${GPR_PROJECT_FILE}"]

[ohenley]

If I may, from the discussion it looks like alr test does not resonate universally on what it means to do.

If it means to do a crate 'sanity-check', as opposed to run a 'test' (suite or else) over the code content of the crate, maybe something along the lines of alr check_sanity, alr check_crate, etc would be more self explaining.

[Lucretia]

Just having a standard way of doing tests, whether that's with aunit or ahven or something else, needs to be implemented. I've banged my head on that wall far too long. Look at how Dune does it, it's really simple in Dune.

[dereknance]

I will echo @ohenley's and @Lucretia's sentiment, as far as the de facto means of running tests for a project is concerned. For example:

• Rust - cargo test
• Zig - zig test
• Go - go test
• JavaScript - npm test
• C#/F#/etc. - dotnet test
• Java - mvn test

cargo's subcommand reigns supreme in my eyes, as it runs unit, integration, and documentation tests in one fell swoop.

Except for zig, each of the programs listed above also act as a package manager with the appropriate subcommand. I think alr could certainly join their ranks if alr test's current behavior was instead available as something along the lines of alr publish --test or --validate, whereas alr test would build and run tests in a project's test/ directory. I won't try to elucidate the interface/structure/format of the contents of that directory for now, as that probably gets a bit too far into the weeds.

[mosteo]

Many things have changed in alr 2.0 since this discussion started, so I will summarize:

• alr test now tests the current crate, so that's in line with expressed expectations
• The default test action is alr build --release
• The default can be overridden by defining an action of type = "test".
• Over time, we have settled on a recommendation which is having a nested crate to do the testing
  • Then, the test action should be given as alr -C <test_crate_dir> run.
  • This way all special dependencies and needs are confined to the nested crate.

What is missing and we could easily do, is to make the above happen during alr init, so any newly created crate has the test infrastructure in place.

In short, we can improve that nested test crate over time, but we could start by creating it by default with what is currently recommended.

Thoughts?

[Fabien-Chouteau]

We always try to look at other package manager when implementing stuff in Alire. I woud be cautious on this particular, especially with Rust, Zig, and Go as these languages have built-in support for unit testing. Ada doesn't have this and therefore there is no standard way to write and run tests.