search

ocharles / tasty-ant-xml

A tasty ingredient to output test results in XML, using the Ant schema. This XML can be consumed by the Jenkins continuous integration framework.
https://hackage.haskell.org/package/tasty-ant-xml
Other
9 stars 14 forks source link

When there are many cabal defined test suites not all output is written to XML. #27

Open jfdm opened 6 years ago

jfdm commented 6 years ago

When using tasty-ant-xml in a project with multiple test suites defined in cabal the output of the last test suite is returned to the specified 'xml' file. This is because when we pass an option to the test suite the same option is passed to all cabal defined test suites.

For example, take the following repo jfdm/a-tasty-problem. If we invoke the test suite using stack as follows:

stack test --test-arguments="--xml=report.xml"

Then only a single 'report.xml' file is generated.

Similar problems can be found with any Tasty Ingredient that asks for an option to be supplied. For example Tasty.HTML and Tasty.States I don't know if this is something that should also be reported upstream to the core tasty package as a caveat about ingredient design, after all tasty (and the ingredients) are working as 'expected'.

jfdm commented 6 years ago

For discussion, I have a local fix that names the report after the cabal test suite executable name. This would be a breaking change to how the ingredient works, but think it is a suitable solution.

jfdm commented 6 years ago

My suggestive fix has been made available online:

https://github.com/jfdm/tasty-ant-xml/tree/name-report-after-test-suite-exe

If this is acceptable, then let me know what else needs changed aside from version numbers, to get this merged.

ocharles commented 5 years ago

This is a bigger problem, as you say. It would be interesting to get @feuerbach's input on this one.

UnkindPartition commented 5 years ago

From my POV, everything is working as expected—including tasty, tasty-ant-xml, and cabal.

cabal provides a shortcut to run many test suites with a single command, passing the same set of options to all suites. Sometimes you don't want the options to be the same—in which case you don't use this shortcut and instead run the suites individually, passing a different --xml to each.

That said, @jfdm your patch does provide an interesting alternative solution (naming the report after the executable). If @ocharles you decide to merge it, I'd recommend keeping the old option as it was (lest people spend time figuring out why their CI setup suddenly broke) and adding the switch as a new option, off by default.

ocharles commented 5 years ago

I agree.

I was thinking the best option might be to have a "file name template" option, where people can splice in the current executable name.

jfdm commented 5 years ago

@feuerbach @ocharles No disagreements from me.

I don't know if I will have time to help/improve this patch in the near future as I have to get on with other things. I agree that the idea of a 'file template' and keeping old behaviour would be beneficial.

ocharles commented 5 years ago

Ok. I'm happy with some type of templating addition, but unfortunately won't be able to add it myself. I don't use this library anymore, so can only offer essentially maintenance. I will happily accept a PR though. @jfdm, you may be best just using your fork for now.