Open sarbogast opened 10 years ago
specs like the whole code, or just names?
Just the text in the describe, context and it clauses.
Kind of like rspec --format documentation
, right? Given the following spec:
describe Car do
context 'when driving down the highway' do
it 'drives fast ' do
end
context 'but behind a crown vic' do
it 'drives slow' do
end
end
end
end
RSpec's documentation formatter outputs:
Car
when driving down the highway
drives fast
but behind a crown vic
drives slow
Finished in 0.00092 seconds
2 examples, 0 failures
We can add this sort of functionality along with custom output formatting, which is part of the roadmap for Kiwi 3.0 (see issue #306).
I have an open RFC for output formatters issued as pull request #449--it would be trivial to add a documentation formatter to that branch. If you'd like to try it out, @sarbogast, I can send you some instructions once I've added a documentation formatter. :neckbeard:
Not sure if this is a testing framework responsibility tho; XCPretty can generate you a nice HTML, but it should be fairly easy to add Markdown output as well
If it's a question of which formatters to include with a testing framework such as Kiwi, then I agree that we can't include them all. I'd be fine with just including JSON and plain text. Users can create and distribute their own formatters for XML, HTML, Markdown, a progress bar, nyancat format, or whatever.
However, I think a testing framework absolutely is responsible for its own output.
Of course, everyone wants different output. Some people want human-readable output, some want XML. @sarbogast wants markdown. Tools like XCPretty are necessary because xcodebuild
doesn't provide a way to format its output. This forces developers to build a parser that relies on the raw output. By not providing an "official" method, Apple can either:
xcodebuild
.I think it's pretty obvious which of the two Apple will choose.
Luckily, we can do better.
RSpec, for example, provides an "official" hook for custom formatting. RSpec provides users with an explicit contract. This contract guarantees that RSpec will send custom formatters a well-defined set of messages during test execution. Formatters can print whatever they like based on those messages. If the messages ever need to change, RSpec can gradually deprecate them. This makes maintainers and users happy: users know their formatters won't break, and maintainers can change the output of built-in formatters without worrying about breaking tools like XCPretty.
While it's certainly possible to parse RSpec output by piping it to an external tool, RSpec provides a more practical solution for doing so. In Kiwi's case, we can add the capability for custom formatting without changing the output XCPretty relies on. In other words, we can maintain that implicit contract while providing a better, more explicit one.
It would be really nice to be able to export specs as a markdown document so that it's easier to communicate to business people what developers are verifying.