search

skaliber / mb-unit

Automatically exported from code.google.com/p/mb-unit
0 stars 0 forks source link

Tests created through static test factory are run, but do not appear in the test report summary. #523

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. Create a test fixture with some statically compiled tests and some
dynamically generated ones (I used StaticTestFactory attribute). Make sure
one dynamically generated test fails.
2. Run the tests.
3. Open the report.

What is the expected output? What do you see instead?
Note, that the report status is red, but all the tests in the summary are
green! This is because the summary does not show the dynamically generated
tests. One has to browse the details section to look for the failed test.

What version of the product are you using? On what operating system?
3.1 build 287
Windows XP

Please provide any additional information below.
I have source code, which clearly demonstrates the problem. This is a
fairly complex project, which I use to evaluate various MbUnit aspects.
It consists of three top level directories - 3rd_party, common and windows.

The tests can be run by changing to the windows\Infra folder and running
msbuild Master.proj (which also compiles the source code).

The test report is Shunra.Infra.Test.Results.mht and it is created in the
Infra folder.

Here is the link for the 3rd_party:
http://groups.google.com/group/gallio-dev/web/3rd_party.zip - it contains
the binaries in Debug and Release.

The source code itself is attached to this issue.
You will need 7zip to unzip the files.

Original issue reported on code.google.com by mark.kha...@gmail.com on 14 Sep 2009 at 7:20

Attachments:

GoogleCodeExporter commented 8 years ago 
Correction. The problem occurs when the dynamically created and statically 
compiled
tests share the same test fixture type, which is the case in the attached 
source code.

Original comment by mark.kha...@gmail.com on 14 Sep 2009 at 9:00

GoogleCodeExporter commented 8 years ago

Original comment by jeff.br...@gmail.com on 29 Sep 2009 at 1:40

GoogleCodeExporter commented 8 years ago 
Yann, can you look into this issue?  I'm not sure I understand exactly what the
problem is.  Sounds like a glitch in the report presentation.

Original comment by jeff.br...@gmail.com on 25 Oct 2009 at 2:58

GoogleCodeExporter commented 8 years ago

Original comment by Yann.Tre...@gmail.com on 17 Jul 2010 at 8:12

GoogleCodeExporter commented 8 years ago 
Better late than never... I have run the following test case which should 
reproduce the problem Mark describes, but the output report looks fine to me 
(see attachment). Did I miss something? Or is it possible that a previous fix 
in v3.2 has solved the subject issue in the same time...

[TestFixture]
public class StaticTestFactoryAttributeTest
{
  [Test]
  public void SimpleTestMethod1()
  {
  }

  [Test]
  public void SimpleTestMethod2()
  {
    Assert.Fail();
  }

  [StaticTestFactory]
  public static IEnumerable<Test> CreateStaticTests()
  {
    yield return new TestCase("StaticTestMethod1", () =>
    {
    });

    yield return new TestCase("StaticTestMethod2", () =>
    {
      Assert.Fail();
    });
  }
}

Original comment by Yann.Tre...@gmail.com on 18 Jul 2010 at 2:36

Attachments:

GoogleCodeExporter commented 8 years ago 
Hi Yann.
Thanks for your work.
I have just checked with my build (Version 3.1 build 514) and the reports seem 
OK. I guess much have changed between this build and build 287, when I 
originally reported the issue.

Original comment by mark.kha...@gmail.com on 18 Jul 2010 at 7:28

GoogleCodeExporter commented 8 years ago

Original comment by Yann.Tre...@gmail.com on 18 Jul 2010 at 8:04

GoogleCodeExporter commented 8 years ago 
:)

Thank you for the info, Mark.

Original comment by Yann.Tre...@gmail.com on 18 Jul 2010 at 8:05

GoogleCodeExporter commented 8 years ago 
Hi.
I hate to bug you again, but there is still something wrong with the report 
engine. I am attaching a complete HTML report, where you can see the problem.

I am also attaching the images of the Summary and Details sections. What I 
cannot understand is what is the logic of what goes where. Note that the test 
GetOfflineSamplesSanity has failed, but it does not appear in the Summary! It 
appears in the Details section. Some tests appear in both sections, while 
others just in the Details section.

It looks buggy.

Thanks.

Original comment by mark.kha...@gmail.com on 2 Aug 2010 at 2:09

Attachments:

GoogleCodeExporter commented 8 years ago 
The summary section displays the parent nodes that contain at least one child 
test.

That's why you see RunDummyProtocolFlowEcho, it contains a child test 
RunDummyProtocolFlowEcho({NetworkCatcher.Entities.Contract.IEndpoint #1 
(MyEndpoint)}); But not GetOfflineSamplesSanity.

However, I agree, it's a bit confusing.
We could certainly improve that by changing a bit the contents of the summary 
section.

Maybe we could show only, or include the nodes in relation with a failing test.
Or just exclude the parents of data-driven tests.
Any suggestion?

Original comment by Yann.Tre...@gmail.com on 5 Aug 2010 at 7:59

GoogleCodeExporter commented 8 years ago

Original comment by Yann.Tre...@gmail.com on 5 Aug 2010 at 8:05

GoogleCodeExporter commented 8 years ago 
I agree. In the condensed report, the summary must either be empty (all passed) 
or every single item in it must be red and all the red items must be present 
there. Otherwise, it kills the purpose of the summary.

Original comment by mark.kha...@gmail.com on 6 Aug 2010 at 4:11

GoogleCodeExporter commented 8 years ago 
Issue 771 has been merged into this issue.

Original comment by Yann.Tre...@gmail.com on 15 Nov 2010 at 7:16

GoogleCodeExporter commented 8 years ago

Original comment by Yann.Tre...@gmail.com on 14 Jun 2011 at 5:53