MemberResourceTest.java's memberResource is NOT Always Assigned

[catch-point]

The @BeforeSuite needs to be replaced with @BeforeTest or used with a static method.

When running a test suite many of the MemberResourceTest tests are incorrectly skipped. This incorrect behaviour is caused because many test instances have memberResource set to null and never run the setup/tearDown methods. Because of the @BeforeSuite annotation on the setup method, it is only run on a single test instance (that might not be reused) and all other MemberResourceTest instances are then skipped. Either memberResource should be changed to a static variable (so all MemberResourceTest instances share the same value) or the annotation should be changed to @BeforeTest, so each instance gets it own value.

[spadgett]

@jamesrdf, how are you running the tests? I only see one MemberResourceTest instance getting created. I tried in Eclipse using the TestNG plugin and on the command line.

[catch-point]

From the command line: $ mvn clean package $ java -jar target/ldp-testsuite-0.2.0-SNAPSHOT-shaded.jar --indirect --server 'http://localhost:8080/ldp/?describe'

49 Skips (including all MemberResource) using the master branch. 28 Skips when I change the annotations to @BeforeTest/@AfterTest and reset memberResource to null in the tearDown.

java version "1.7.0_65" OpenJDK Runtime Environment (IcedTea 2.5.2) (7u65-2.5.2-3~14.04) OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

[catch-point]

Results are not always consistent, sometimes master branch works sometimes it doesn't.

[catch-point]

This comment might be useful: http://stackoverflow.com/questions/3407514/does-testng-guarantee-beforesuite-methods-are-executed-before-beforetest-metho/3411597#3411597

[spadgett]

@jamesrdf, the problem in the Stack Overflow post is @BeforeSuite annotation is on a method in a shared parent class. That's not the case with MemberResourceTest.setup().

I'm not saying there isn't a problem, but I'm not convinced we have it right. For what it's worth, I get consistent results running against my server.

java -jar target/ldp-testsuite-0.2.0-SNAPSHOT-shaded.jar --server http://ldpjs.mybluemix.net/r/bc --basic

Are you sure that the POST request isn't sometimes failing and sometimes succeeding against your server? Maybe running the indirect container tests first changes the result?

[catch-point]

With the patch the results are very consistent.

Wireshark reveals no unexpected failed POST responses.

[catch-point]

After further testing it appears that the setup method is sometimes not called (at all) in the test suite.

[spadgett]

@jamesrdf I can reproduce against my server if I switch to --indirect. The setup() method never gets called.

[spadgett]

It seems that if I remove the tests marked "enabled = false" from IndirectContainerTest, MemberResourceTest.setup() is called again. @jamesrdf, can you confirm that works for you as well?

[catch-point]

Sometimes it is still not being called when those tests are removed, but perhaps failing less frequently.

[catch-point]

I think the problem is that these methods override a super class that is common to both the container and member tests. Test-NG might something think they were already called (when it invoked LdpTest#setup on the IndirectContainerTest instance).

If I rename the setup tearDown methods to a unique name, I get (the desired) consistent results.