search

elm-explorations / test

Write unit and fuzz tests for Elm code.
https://package.elm-lang.org/packages/elm-explorations/test/latest
BSD 3-Clause "New" or "Revised" License
236 stars 40 forks source link

capture Debug.log and present it with a failing test #154

Open BrianHicks opened 3 years ago

BrianHicks commented 3 years ago

When I'm trying to figure out why a test is failing, I'll often put Debug.log statements into the things I'm modifying to see what's up. If I run my whole test suite after that, I get way more log messages than I want, with no indication of which ones are actually relevant to the test failure I'm trying to debug.

I'd really like it if, instead, elm-test would capture stdout and associate it with a given test run. A few other test frameworks I've used (py.test maybe? but it's been a while) do this and it's super nice.

If I could have a pony, I'd also like this for fuzz tests. I'd like to only see the debug messages from the final shrunk failure case!

BrianHicks commented 3 years ago

current workaround: slapping an only on the relevant test. I don't like this as much because it makes me choose between useful debug output and running the whole suite to make sure I didn't break something else with a change.

harrysarson commented 3 years ago

See https://github.com/rtfeldman/node-test-runner/issues/400 and (slightly less relevant) https://github.com/rtfeldman/node-test-runner/issues/327. Ava does this really nicely and we should too!

drathier commented 3 years ago

I've suggested this in the past too, specifically for fuzz tests, where shrinking duplicates the debug.log output massively. I definitely think it's doable, by passing a context record to the test/fuzz lambdas, where one field could be a log function, which is a no-op until a test fails, or a failing fuzz test has been shrunk enough. This should also allow sorting output by test.

drathier commented 3 years ago

I did a proof of concept at #156