Option to enable per example output from JSON formatter

astley92 commented 1 month ago

Per Example JSON Formatter

The JSON formatter is super useful for dev tools that utilise the output of RSpec as from what I can tell it's the most reliable way to programatically get the results and information about individual examples and the overall run.

It's not so much an issue, more of a nice to have, but it would be nice to have an option to allow the JSON formatter to output a JSON object for each example run progressively as well as the summary at the end of the run (this summary already includes each example but is only output at the end of the run).

This can easily be achieved with a custom formatter or monkey patching the existing JSON formatter and I've used those both in the past which is fine, but I feel it would be a nice option for the core implementation to offer out of the box.

I'd be keen to put a PR up if it's worthwhile

Discussion

Apologies if I've missed something and this can already be achieved or if this is not a good idea. No problems closing this if not wanted

If open to a PR, I have a couple notes:

Would need to be opt in, not just update all usage of the existing json formatter.
Would the most acceptable way to do this be an entirely new formatter or just a simple flag to be passed in the command line for json formatter only?

GIST

Just an example/ semi sudo code in case I haven't explained sufficiently.

class RSpec::Core::Formatters::JsonFormatter < RSpec::Core::Formatters::BaseFormatter
  RSpec::Core::Formatters.register self, :example_passed, :message, :dump_summary, :dump_profile, :stop, :seed, :close

  def example_passed(notification)
    return unless per_example_json_enabled

    @output << format_example(notification.example).to_json
  end
end

pirj commented 1 month ago

for dev tools that utilise the output of RSpec output a JSON object for each example run progressively

What are those tools that can consume a stream of JSON?

summary already includes each example but is only output at the end of the run

Sorry, I didn’t quite get the suggestion, to add an intermediate summary section after each example?

If there’s real demand from dev tools that consume RSpec’s output, I’d happily consider accepting such a PR. But so far I’m more inclined towards a separate gem with such a custom formatter.

JonRowe commented 1 month ago

I personally would encourage this being released as a seperate gem, e.g.rspec-jsonl-formatter or something.

This is really a whole different formatter (and I certainly would implement it that way if this was in core) to the current json one even if it shares some similarities.

I'm going to close this for now as I don't think it should be in core, but if you find you would like to extract some of the existing formatter into a seperate module for printing etc thats public api I'd be happy to review that PR

rspec / rspec-core