aiiie / cram

Functional tests for command line applications
GNU General Public License v2.0
199 stars 51 forks source link

Add an option to exit on first failue #26

Open jan-matejka opened 7 years ago

jan-matejka commented 7 years ago

Add an option to exit on first failue so cram --fail-fast ./tests exits on first failing test case and one can focus only the single test case failing output.

roman-neuhauser commented 4 years ago

This is implemented in Dram (https://git.sr.ht/~rne/dram), see https://git.sr.ht/~rne/dram/tree/master/t/fail-fast/opt.t:

setup::

  $ for x in a b d e; do
  >   printf -->$x '  $ true\n'
  > done

  $ printf -->c '  $ false\n'

test::

  $ dram -Dfv a b c d e
  . a
  . b
  ! c

  tests: 5, skipped: 2, failed: 1
  [1]
aiiie commented 4 years ago

@roman-neuhauser I’ve been kind of too busy with life to think about programming.

I’ve always been kinda intimidated by D, but I wish someone would take over the project and go in whatever direction they like. If you’re interested in extra responsibility for no reason, I’m happy to talk directly!

aiiie commented 4 years ago

And so you know, I might’ve mentioned this in my tests, but my original goal was to be able run the Mercurial test suite’s shell tests. I’m sure their format has significantly diverged in the past few years though so I’d never want to break backwards compatibility for Cram.

I’ve also had the goal for the tests to be runnable with identical output in Bash, Zsh, and Dash. So I recommend testing your tests in those shells.

roman-neuhauser commented 4 years ago

@roman-neuhauser I’ve been kind of too busy with life to think about programming.

I’ve always been kinda intimidated by D, but I wish someone would take over the project and go in whatever direction they like. If you’re interested in extra responsibility for no reason, I’m happy to talk directly!

Hey!

Sure, my email is in my profile.

roman-neuhauser commented 4 years ago

And so you know, I might’ve mentioned this in my tests, but my original goal was to be able run the Mercurial test suite’s shell tests. I’m sure their format has significantly diverged in the past few years though so I’d never want to break backwards compatibility for Cram.

I had a look at Mercurial's tests a few weeks ago and it seemed to have gone some ways indeed. I can't parse the second sentence, sorry. :) Anyway, to riff on the theme of BC: I've been using Cram for many years, and loved the simplicity, aesthetics, and stability of the test format all the way. I take stability of interfaces very seriously (http://rants.sigpipe.cz/durable.html), and should Dram become the next version of Cram, things like escaping metacharacters in (glob) lines will have to be figured out.

I’ve also had the goal for the tests to be runnable with identical output in Bash, Zsh, and Dash. So I recommend testing your tests in those shells.

Dang, NOCLOBBER strikes again! Thank's for the headsup, I have it fixed locally and will push shortly.

for s in sh bash dash mksh posh zsh; do DRAM_SHELL=/bin/$s make check; done

now passes.

roman-neuhauser commented 4 years ago

Oh, if you're saying that running Mercurial's current test suite is a non-goal for you, then rest assured that it's the same for me.