search

marick / Midje

Midje provides a migration path from clojure.test to a more flexible, readable, abstract, and gracious style of testing
MIT License
1.68k stars 128 forks source link

XML JUnit report format #121

Closed AlexBaranosky closed 11 years ago

AlexBaranosky commented 12 years ago

From the Midje Google group: "Has anyone succeeded in getting Midje--preferably from lein midje--to output JUnit-compatible XML files for use with Hudson/Jenkins?

I found this thread from January 2011, but have not been able to find any examples of anyone taking it beyond that discussion: http://groups.google.com/group/midje/browse_thread/thread/710755b8deea5e4.

Thanks, Greg"

AlexBaranosky commented 12 years ago

Hi Greg,

Do you have a link to what the JUnit-compatible XML format looks like? It seems to me that we just need a way to specify an alternate report format, which should be relatively straightforward to implement -- we'd just need to map Midje's failure model over to JUnits and hope that they map fairly naturally.

The report-strings multimethod dispatches on the type of the report map. All methods are here: https://github.com/marick/Midje/blob/master/src/midje/internal_ideas/report.clj#L62

The approach I'm thinking of is to write a report-junit-xml multimethod, and swap that in if the proper configuration is in place. Then the simplest thing might be to pipe stdout to a file.

The more interesting question to me is how Midje should be handling these types of configurations. Should there be a midje.config? Should there be dynamic vars for the configurable bits? Should the midje.config set! those dynamic vars?

Alex

AlexBaranosky commented 12 years ago

More from @gregspurrier -

"Hi Alex,

Here's an answer from StackOverflow that I think has enough info: http://stackoverflow.com/questions/4922867/junit-xml-format-specification-that-hudson-supports

I'm going to ponder your other questions for a bit before responding.

Greg"

AlexBaranosky commented 12 years ago

Hi @gregspurrier, I have started to adjust the namespace structure so that we can more easily add a new report formatter. (https://github.com/marick/Midje/commit/cf2f0f1375a11d3d8db08575a71fce6b4f51de4f)

The idea would be to implement something like https://github.com/marick/Midje/blob/master/src/midje/ideas/reporting/string_format.clj perhaps calling it junit_xml_format.clj . All that would be needed is a function or multimethod that writes out XML strings appropriate to the maps sent into report.clj.

I've also think we should bring the report summary code into Midje source, rather than have it be in lein-midje. (https://github.com/marick/lein-midje/blob/master/src/leiningen/midje.clj#L13) That way, when writing different report formats, all of the code for them would be located in the same place, and maintaining lein-midje would avoid getting out of hand.

I plan to apply this refactoring first before adding any xml reporting functionality.

If you are interested in TDDing up a namespace parallel to string_format.clj called junit_xml_format.clj that could be started in parallel. Just look to t-report.clj for inspiration as to how to go about unit testing it.

AlexBaranosky commented 12 years ago

@gregspurrier added mostly empty file to add new format to in above commit ^^^

gregspurrier commented 12 years ago

@AlexBaranosky Thanks for getting the ball rolling. I'll see if I can find some time to work on it tomorrow.

AlexBaranosky commented 12 years ago

I'm still thinking about how to move the summary code from lein-midje to midje... If I move it, it would mean I need to handle backwards compatibility issues in the plugin... Need hammock time on that. Maybe just adding extra stuff to the plugin is the right immediate solution.

AlexBaranosky commented 12 years ago

Any thoughts on how to handle backwards compatibility here? If I move the summary reporting code out of lein-midje, then anyone not using the newest version of Midje won't have access to the summary code.

When I say it that way, it seems clear that I probably need to leave the summary code in the lein-midje plugin for some time, but first check to see if there is an overriding summary format function in Midje to use instead, if present.

AlexBaranosky commented 12 years ago

I'm not sure how to call into Midje code from the plugin though -- how to make sure that the version of Midje used by the plugin is the same as the version being used in the user's test code.

bcmreed commented 11 years ago

Hey guys, anyone made any further progress on this? Being able to get midje -> junit output into our CI (teamcity) would be a huge win.

marick commented 11 years ago

I'll be working a quarter time on Midje for a while. I'll take a look at this soon.

marick commented 11 years ago

I'm embarking on a perhaps-foolish rewrite of the midje reporting code, which has suffered from (1) a desire to use the core.test reporting as a foundation (when it should be a side-feature) and (2) tardiness in taking control of the evolution of Midje from (basically) a stream of individual checks to something with much more structure (facts within facts, streams of fact checks within namespaces, etc.)

marick commented 11 years ago

1.5-alpha7 has a plugin interface that can be used for this.

jonpither commented 11 years ago

Am also wondering what the latest is with this? Has someone started work on it? If not is help wanted?

josephwilk commented 11 years ago

On 22 Mar 2013 15:54, "jonpither" notifications@github.com wrote:

Am also wondering what the latest is with this? Has someone started work on it? If not is help wanted?

Yes I did start on this here: https://github.com/josephwilk/Midje/compare/junit-formatter

There is also a TAP formatter in development.

My goal is to get Jenkins to understand our midje test results. So I'm happy with TAP or junit formatter.

The formatter api is still relatively new so it has some kinks we are looking at.

Would you need a specific junit formatter or would something Jenkins understand suffice?

— Reply to this email directly or view it on GitHub.

jonpither commented 11 years ago

Hi Joseph. Our immediate want is for Jenkins to understand the test output. What's the status of your plugin - is it going to be merged into Midje soon? If so then great :-)

Jon.

josephwilk commented 11 years ago

So the proof of concept is ready. I have this generating junit xml for Midje and some local projects. Jenkins is correctly parsing the generated xml report.

https://github.com/josephwilk/Midje/tree/junit-formatter

I will start work on the real thing for a pull request.

If you are eager to start using this now you can use this: https://github.com/josephwilk/midje-junit-formatter

On Mon, Mar 25, 2013 at 9:51 AM, jonpither notifications@github.com wrote:

Hi Joseph. Our immediate want is for Jenkins to understand the test output. What's the status of your plugin - is it going to be merged into Midje soon? If so then great :-)

Jon.

— Reply to this email directly or view it on GitHubhttps://github.com/marick/Midje/issues/121#issuecomment-15382679 .

marick commented 11 years ago

In 1.6