search

typelevel / weaver-test

A test framework that runs everything in parallel.
https://typelevel.org/weaver-test/
Other
46 stars 8 forks source link

Machine readable output #81

Open zainab-ali opened 2 weeks ago

zainab-ali commented 2 weeks ago

This issue was copied over from: https://github.com/disneystreaming/weaver-test/issues/716 It was opened by: caenrique

Would it be possible to have the tests results written as json? This would make tooling integration much simpler

zainab-ali commented 2 weeks ago

This comment was copied over from: https://github.com/disneystreaming/weaver-test/issues/716#issuecomment-1824143874 It was written by: Baccata

Can you elaborate on the context of the question ? Weaver doesn't come with a CLI, but integrates with Scala build tools (sbt, mill, scala-cli) via the sbt test-interface https://github.com/sbt/test-interface.

How are you intending to run the tests, such that results written in json would be relevant ?

zainab-ali commented 2 weeks ago

This comment was copied over from: https://github.com/disneystreaming/weaver-test/issues/716#issuecomment-1824244147 It was written by: caenrique

Yes, sorry, I'll give more context

I'm using Sbt and I'm trying to integrate test results with Neovim. What this means for me is being able to filter failing tests, go to test location and show the output of the failed test.

To do this I need test result, test name, file, line of the test, output.

I've found a way of doing this to some extent by using an Sbt plugin which extract some of that information using sbt.TestListener which depends on the Sbt test-interface as you mention.

an example of what I'm getting back:

{
  "timestamp": "2023-11-12T21:17:28",
  "status": "SUCCESS",
  "durationWithSetup": 0.015,
  "duration": 0.087,
  "suite": "com.package.SuiteName",
  "test": "this test does something",
  "test-timestamp": "2023-11-12T21:17:38",
  "errorMessage": "",
  "stackTrace": ""
}

Ultimately this data is being extracted from a sbt.testing.Event

There is an important bit of information missing though, which is the test output (e.g. why the expectations failed, diff outputs, etc) which I believe is produced using a logger.

Now, I'm not familiar with the implementation here, so this is me mostly trying to find a good way to implement this. The way I was thinking to work around this is by parsing the test output, but it's a bit tricky and would be much easier if we could change the logger format to be json by using a config value or something like that.

Sorry for the rant and if you think there is a simpler way of achieving this, I would greatly appreciate the feedback :)

zainab-ali commented 2 weeks ago

This comment was copied over from: https://github.com/disneystreaming/weaver-test/issues/716#issuecomment-1889586143 It was written by: Baccata

Sorry for getting back to you only now. Could you provide an example json payload you get for a failed test ?

zainab-ali commented 2 weeks ago

This comment was copied over from: https://github.com/disneystreaming/weaver-test/issues/716#issuecomment-1994273159 It was written by: kubukoz

uninformed question, isn't this sort of thing covered by BSP?