Open erikd opened 6 years ago
That's a confusing doc message. --no-run-benchmarks
should probably give an error or warning if the benchmarks weren't going to be run or built anyway. To get benchmarks building, you'd use stack test --bench --no-run-benchmarks
.
Wow, that really is not very intuitive.
--no-run-benchmarks
should probably give an error or warning if the benchmarks weren't going to be run or built anyway.
I cannot agree strongly enough.
Alternatively, stack bench --no-run-benchmarks
. I agree that there should be a warning for this.
There actually seem to be the same issue with:
stack build --no-run-tests
which doesn't actually build the tests.
Hmm, yes, I can certainly see the convenience of --no-run-tests
also implying --test
(and the same for --no-run-benchmarks
). That seems better than a warning.
I just discovered a similar one: --git-branch
for stack upgrade
doesn't imply --git
, see: https://github.com/commercialhaskell/stack/issues/4125#issuecomment-402559558.
I have a question regarding this bug. I'm aware that stack test
and stack bench
are just aliases to stack build --test
and stack build --bench
. Underneath they both call build
, so obviously they share same set of flags.
However if we look semantically at them, then we see that stack test
and stack bench
are totally separated from each other conceptually. If I have a broken benchmark (does not compile) and I run stack test
, the benchmark will not be neither compiled, build or run. So, when running tests, adding a flag --no-run-benchmark
does not make sense because they would not run anyway. I understand that the flag tells "Disable running of benchmarks. (Benchmarks will still be built.)" - but they are already disabled and NOT building, because the user tries to run tests only.
I think the main problem is that test
and bench
are just aliases. That they share common set of flags - all being part of the build
command. If test
and bench
were not aliases but first class commands, then test
would never have --no-run-benchmark
flag, correct?
CC: @snoyberg
Not quite. It's perfectly valid to run stack test --bench, for example. You can email multiple optional components at once.
On Thu, Jul 5, 2018, 10:30 AM Pawel Szulc notifications@github.com wrote:
I have a question regarding this bug. I'm aware that stack test and stack bench are just aliases to stack build --test and stack build --bench. Underneath they both call build, so obviously they share same set of flags. However if we look semantically at them, then we see that stack test and stack bench are totally separated from each other conceptually. If I have a broken benchmark (does not compile) and I run stack test, the benchmark will not be neither compiled, build or run. So, when running tests, adding a flag --no-run-benchmark does not make sense because they would not run anyway. I understand that the flag tells "Disable running of benchmarks. (Benchmarks will still be built.)" - but they are already disabled and NOT building, because the user tries to run tests only.
I think the main problem is that test and bench are just aliases. That they share common set of flags - all being part of the build command. If test and bench were not aliases but first class commands, then test would never have --no-run-benchmark flag, correct?
CC: @snoyberg https://github.com/snoyberg
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/commercialhaskell/stack/issues/4045#issuecomment-402630997, or mute the thread https://github.com/notifications/unsubscribe-auth/AADBB4sJx4JgM95FGN_YY7ofCoXMaBURks5uDcCEgaJpZM4UQyM_ .
Or maybe I make a mistake by treating (conceptually) test
and bench
as separate commands. They are just "flags" that trigger some behaviours on build (like "build test", "run tests", "dont build tests"). Then if so, how the stack system should determine which flag has higher priority, given that there are two flags that have opposed functionalities - like --test --no-run-benchmark
, where --test
is saying "dont build benchamrks" and --no-run-benchmark
is saying "build benchmarks".
--test
doesn't say "don't build benchmarks", it says "build tests." Same with --bench
. You can do both of these at the same time.
I have all I need, thx :)
Using stack version 1.7.1 (current latest).
General summary/comments (optional)
The
--no-run-benckmarks
option to thetest
command does not work as described in thestack test --help
output.Steps to reproduce
This passes:
but then this:
fails with a build error (in one of the benchmarks) that should have been caught by the
stack test --no-run-benchmarks
command.Expected
In this case, since there actually is a build error when building the benchmarks,
stack test --no-run-benchmarks
should exit with a non-zero exit code. This is important because we want changes in libraries that break the benchmarks to be caught in CI.Actual
The 'stack test --no-run-benchmarks' command doesn't build the all benchmarks and therefore does not fail the CI build.
Stack version
The git version above is not the git version of stack, but rather the git version of the repository where I keep metadata for creating a Debian package for my own use. My packaging metadata builds the version specified from the Hackage tarball.
Method of installation