Closed zyv closed 7 years ago
@zyv Hey, thank you for the issue.
The failing tests are by design. The idea is that at the moment you provide a specification, the tests and controller skeletons are generated. But at this moment the tests should fail because there is no implementation in controllers yet. That's exactly what we want to see in our scripted tests - after initial artefacts were generated, the tests are red.
Does it make sense?
Hi Slava,
Thank you for the explanation! I can roughly follow what you are saying, but I would argue that it's very confusing to see lots of failures as a result of running scripted tests, so maybe the output could be somehow rearranged to make it more clear what's going on & what's expected and/or document this. However, I can't propose a better solution, because I'm not knowledgeable enough about your code; so feel free to close this issue...
--Yury.
@zyv Hey, thanks for your feedback. One idea could be instead of returning "Not implemented" from freshly generated controllers to return some random generated data. This would make tests pass, but the downside is that the developer cannot distinguish anymore between tests passing because of the proper implementation or just because of this random data returned from the API.
Leaving this issue open to investigate the possibilities further.
Yes, returning random data doesn't sound like a great option at all... Moreover, there is always a chance that the tests will accidentally pass.
I would have actually expected that expected failures are interpreted as success and unexpected successes as failures in the integration tests context, only as I said I have no constructive suggestions on how to implement that.
That's exactly what is happening during the scripted test, I guess log entry prefix [info] [error]
should give a hint that this is a nested test configuration in which outer test expects inner test to fail.
Here is the abridged output of
sbt scripted
, I suspect that the failing test have to do with the output tagged as[error]
: