Investigate long running tests

[stanlemon]

The purpose of this issue to track improvements in our test suite runtime.

I cleaned up and reviewed the output from one of our test runs. There are a handful of test classes that consume the bulk of execution time. I suspect a few factors are at play we should investigate:

1. Setting up the same bootstrap over and over again
2. Using Zookeeper and Kafka where it is not necessary (we could use functional mocks in places where the hard dependency of the service is not fundamental to the test).
3. Testing the same thing in multiple tests.

I got the output below by looking for any test classes that took longer than 10 seconds to execute, and then from there I filtered out any individual test function less than 1 second.

Running com.salesforce.storm.spout.dynamic.kafka.ConsumerTest
Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.404 sec - in com.salesforce.storm.spout.dynamic.kafka.ConsumerTest

Running com.salesforce.storm.spout.dynamic.buffer.MessageBufferTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.612 sec - in com.salesforce.storm.spout.dynamic.buffer.MessageBufferTest
testConcurrentModification[0: com.salesforce.storm.spout.dynamic.buffer.FifoBuffer@5bd03f44](com.salesforce.storm.spout.dynamic.buffer.MessageBufferTest)  Time elapsed: 12.288 sec
testConcurrentModification[1: com.salesforce.storm.spout.dynamic.buffer.RoundRobinBuffer@29626d54](com.salesforce.storm.spout.dynamic.buffer.MessageBufferTest)  Time elapsed: 11.322 sec

Running com.salesforce.storm.spout.dynamic.DynamicSpoutTest
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.741 sec - in com.salesforce.storm.spout.dynamic.DynamicSpoutTest
doTestWithSidelining(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 6.179 sec
doBasicFailTest(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 5.129 sec
testConsumeWithConsumerGroupEvenNumberOfPartitions[0: 0](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 4.543 sec
testConsumeWithConsumerGroupEvenNumberOfPartitions[1: 1](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 4.486 sec
testResumingSpoutWhileSidelinedVirtualSpoutIsActive(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 17.047 sec
testConsumeWithConsumerGroupOddNumberOfPartitions[0: 0](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 5.556 sec
testConsumeWithConsumerGroupOddNumberOfPartitions[1: 1](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 4.572 sec
testResumingForFirehoseVirtualSpout(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 9.443 sec
testReportErrors(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 2.13 sec
doBasicConsumingTest[0: <null>, default](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 2.644 sec
doBasicConsumingTest[1: SpecialStreamId, SpecialStreamId](com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 3.158 sec
testDeactivate(com.salesforce.storm.spout.dynamic.DynamicSpoutTest)  Time elapsed: 1.571 sec

Running com.salesforce.storm.spout.dynamic.SpoutCoordinatorTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.331 sec - in com.salesforce.storm.spout.dynamic.SpoutCoordinatorTest
testAddAndRemoveVirtualSpout(com.salesforce.storm.spout.dynamic.SpoutCoordinatorTest)  Time elapsed: 1.157 sec
testCoordinator(com.salesforce.storm.spout.dynamic.SpoutCoordinatorTest)  Time elapsed: 4.097 sec
testRestartsSpoutMonitorOnDeath(com.salesforce.storm.spout.dynamic.SpoutCoordinatorTest)  Time elapsed: 20.075 sec

Running com.salesforce.storm.spout.dynamic.VirtualSpoutTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.889 sec - in com.salesforce.storm.spout.dynamic.VirtualSpoutTest
testDoesMessageExceedEndingOffsetWithNoEndingStateDefined(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.095 sec
testAckWithInvalidMsgIdObject(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.046 sec
testUnsubscribeTopicPartition(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.029 sec
testNextTupleIgnoresMessagesThatHaveExceededEndingStatePositionSinglePartition(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.049 sec
testCloseWithCompletedFlagSetToTrueNoSidelineREquestIdentifier(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.038 sec
testCallingOpenTwiceThrowsException(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.062 sec
testNextTupleWhenConsumerReturnsNull(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.033 sec
testDoesMessageExceedEndingOffsetWhenItDoesNotExceedEndingOffset(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.039 sec
testDoesMessageExceedEndingOffsetWhenItEqualsEndingOffset(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.063 sec
testCloseWithCompletedFlagSetToFalse(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.057 sec
testNextTupleReturnsNullWhenFiltered(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.067 sec
testFailWithInvalidMsgIdObject(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.033 sec
testCloseWithCompletedFlagSetToTrue(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.043 sec
testDoesMessageExceedEndingOffsetWhenItDoesExceedEndingOffset(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.065 sec
testDoesMessageExceedEndingOffsetForAnInvalidPartition(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.034 sec
testNextTuple(com.salesforce.storm.spout.dynamic.VirtualSpoutTest)  Time elapsed: 6.053 sec

[Crim]

We also need to watch out for things like this: https://github.com/salesforce/storm-dynamic-spout/pull/60

I'm guilty of falling into the same trap with the PresistenceAdapter tests previously. There may be a few other spots where we aren't properly closing out background/async threads in tests. These start to pile up as essentially leaks (memory and CPU)