Analyse Test Results?

[ebebbington]

Description

This is a question.

Are there plans on added support for being able to 'analyse' tests results? For example:

const result = Deno.test("My test", () => {
  assertEquals(true, true)
  assertEquals(false, true)
})
console.log(result)
// {
//   failures: [
//     {
//      line: 3,
//      assertion: "assertEquals",
//      output: {
//        expected: false,
//        actual: true
//      }
//     }
//   ],
//   passes: ...
// }

Use Case

Note: of course this is my own use case, but this feature may provide more utility, and i'm not saying just because it would be useful to me, i expect it to be implemented

I have an idea for adding a GUI for a browser automation and testing tool (similar to Cypress.io), and i've been playing around with the idea of displaying test results for each test. I think the ability to analyse tests results would be sufficient in this scenario. The only current way i can think of doing this, is running the test inside a sub process, getting the stdout/stderr, and filtering the result so I can end up with the - "..."\n+ "..." blocks

[nayeemrmn]

In the past we've discussed hooks for the test runner looking something like this:

Deno.test("My test", () => {
  assertEquals(true, true)
  assertEquals(false, true)
});

Deno.setTestHook((testMessage) => {
  if (testMessage.testStart != null) {
    // ...
  } else if (testMessage.testEnd != null) {
    // ...
  }
  return testMessage;
});

But no good use case has arisen for custom test reporters which are enclosed in the same JS context as the tests.

The best solution to the use case mentioned here is a deno test --json, which is inevitable anyway.

[ebebbington]

I agree, seems like deno test --json will fit the usecase perfectly

[David-Else]

VS Code team are working on integrating a testing API and UI. They already have a prototype: https://github.com/microsoft/vscode/issues/107467

[caspervonb]

Been playing around with a few iterations of bringing the test runner over to rust, which involves custom test reporters (e.g deno test --json).

Coverage is next in my queue tho.

[ebebbington]

Will probably close this issue then - is there an issue for the --json feature?

[caspervonb]

Will probably close this issue then - is there an issue for the --json feature?

I guess this is it :D I've mentioned it in various issues, discord and #7915

[ebebbington]

Ok cool thanks @caspervonb :) To reference what i mentioned. on discord, I think it'd be great if there was a way to get what files were tested. - maybe this could be part of --json? so for example: deno test tests/<some_dir> would display which files were tested as well, as opposed to just the test cases

[caspervonb]

I think it'd be great if there was a way to get what files were tested. - maybe this could be part of --json? so for example: deno test tests/ would display which files were tested as well, as opposed to just the test cases

Yeah would be an array containing filenames and arrays of test results, exposing the filename is a big part of why I want the rewrite.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[caspervonb]

Not stale; been experimenting and will do a proper run at this once I've wrapped my current PRs.

[nayeemrmn]

I've been saying deno test --json, but I think it should be deno test --ndjson that streams the test events (testStart, testEnd, complete) in newline-delimited JSON format.

Ref https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON, http://ndjson.org/.

[halvardssm]

I tried to find it in the documentation, but could not see any reference to a json reporter. Did this ever get implemented, or what is the status for this issue?

[bartlomieju]

@halvardssm we don't have JSON reporter. You can use --reporter=junit or --reporter=tap for a structured output.