search

spideruci / tacoco

Integrated Software Analysis Framework
https://github.com/spideruci/tacoco
MIT License
8 stars 7 forks source link

Ignore container events for the unifiedtestlistener #112

Closed kajdreef closed 4 years ago

kajdreef commented 4 years ago

Made a change that all the container events are filtered out, and we only care about the actual tests.

I am not sure if this is something we care about, however, the JUnit platform has events for two types of components: (1) a container, and (2) tests. For a testlistener we probably care only about the latter.

Do you have any thoughts on this?

VijayKrishna commented 4 years ago

What is a container?

kajdreef commented 4 years ago

What is a container?

I coudn't quickly find a formal definition of how the junit platform defines a container, but as far as I can see is that a container can hold more containers or tests. So containers hold something, but are not actual tests and help us determine what testrunner to use and where the tests are located. For example, when running my example testrunner it generates the following output:

Entering container: JUnit Jupiter
Entering container: JUnit5Tests
Test was started: testIfTrueIsTrue()
Finished testIfTrueIsTrue() - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Test was started: testIfFalseIsFalse()
Finished testIfFalseIsFalse() - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Leaving JUnit5Tests
Leaving JUnit Jupiter
Entering container: JUnit Vintage
Entering container: JUnit3Tests
Test was started: test_IfFalseIsFalse
Finished test_IfFalseIsFalse - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Test was started: test_IfTrueIsTrue
Finished test_IfTrueIsTrue - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Leaving JUnit3Tests
Entering container: JUnit4Tests
Test was started: testIfTrueIsTrue
Finished testIfTrueIsTrue - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Test was started: testIfFalseIsFalse
Finished testIfFalseIsFalse - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Leaving JUnit4Tests
Leaving JUnit Vintage
Entering container: TestNG TestEngine SPIKE
Entering container: TestNGTests
Test was started: testIfFalseIsFalse
Finished testIfFalseIsFalse - TestExecutionResult [status = SUCCESSFUL, throwable = null]
Test was started: testIfTrueIsTrue
Finished testIfTrueIsTrue - TestExecutionResult [status = SUCCESSFUL, throwable = null]

===============================================
Command line suite
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================

Leaving TestNGTests
Leaving TestNG TestEngine SPIKE
Total of tests started: 8
Total of tests succeeded: 8
Total of tests failed: 0

You can see this as kind of 3 steps that repeat four times for the different test engines:

  1. Entering the container with the correct TestEngine
  2. Entering a class containing tests (still a container)
  3. Start executing the test within the container

I think we can ignore the entering of the containers, because we are only interested in the events that occur when we execute a test, right?

Just from a stylistic standpoint, might be better to do an early return, and keep the nesting to a minimum. Makes it easier to decipher the "happy" path in the code.

I agree, I will change it to early returns.

VijayKrishna commented 4 years ago

I think we can ignore the entering of the containers, because we are only interested in the events that occur when we execute a test, right?

Perhaps. Sounds like it should be fine. But until there is a clear understanding of what this might be, i would keep a watchful eye on this. Keep doing spot checks to see if the number of tests in a project (from your experiments) line up with the number of tests reported by Tacoco.

Also test this with a suite that has Parameterized test cases.

kajdreef commented 4 years ago

I tested it on an parameterized test case, and all the executions show up, see example output below from spiderMath_JUnit4.

{"testCount":11,
"testsIndex":[
  "testAdd[0].[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.AdditionParamTest]/[test:%5B0%5D]/[test:testAdd%5B0%5D(org.spideruci.benchmark.spiderMath_JUnit4.AdditionParamTest)]",
  "testAdd[1].[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.AdditionParamTest]/[test:%5B1%5D]/[test:testAdd%5B1%5D(org.spideruci.benchmark.spiderMath_JUnit4.AdditionParamTest)]",
  "testSuccess1.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest]/[test:testSuccess1(org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest)]",
  "testSuccess2.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest]/[test:testSuccess2(org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest)]",
  "testFail.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest]/[test:testFail(org.spideruci.benchmark.spiderMath_JUnit4.AdditionTest)]",
  "testSuccess1.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest]/[test:testSuccess1(org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest)]",
  "testSuccess2.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest]/[test:testSuccess2(org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest)]",
  "testFail.[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest]/[test:testFail(org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationTest)]",
  "testMulti[0].[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationParamTest]/[test:%5B0%5D]/[test:testMulti%5B0%5D(org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationParamTest)]",
  "testMulti[1].[engine:junit-vintage]/[runner:org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationParamTest]/[test:%5B1%5D]/[test:testMulti%5B1%5D(org.spideruci.benchmark.spiderMath_JUnit4.MultiplicationParamTest)]",
  "end"
],
"sources":[{
  "source": {
    "fullName": "org/spideruci/benchmark/spiderMath_JUnit4/Addition.java",
    "firstLine": 3,
    "lastLine": 11
  },
  "activatingTests": [
    0,
    1,
    2,
    3
  ],
  "testStmtMatrix": [
    [
      false,
      false,
      false,
      true,
      true,
      false,
      false,
      false,
      false
    ],
    [
      false,
      false,
      false,
      true,
      true,
      false,
      false,
      false,
      false
    ],
    [
      false,
      false,
      false,
      true,
      true,
      false,
      false,
      false,
      false
    ],
    [
      false,
      false,
      false,
      true,
      true,
      false,
      false,
      false,
      false
    ]
  ],
  "coverableLines": [
    true,
    false,
    false,
    true,
    true,
    false,
    false,
    false,
    true
  ]
},
{
  "source": {
    "fullName": "org/spideruci/benchmark/spiderMath_JUnit4/Multiplication.java",
    "firstLine": 3,
    "lastLine": 6
  },
  "activatingTests": [
    5,
    6,
    8,
    9
  ],
  "testStmtMatrix": [
    [
      false,
      false,
      false,
      true
    ],
    [
      false,
      false,
      false,
      true
    ],
    [
      false,
      false,
      false,
      true
    ],
    [
      false,
      false,
      false,
      true
    ]
  ],
  "coverableLines": [
    true,
    false,
    false,
    true
  ]
}]}

Currently, "end" also shows up, I am not sure why we are setting a sessionID "end", It seems you have asked a similar question before #3. Did you ever find an answer? In the mean time, I will merge these changes.