[KIEKER-1283] Split Unit and Integration Tests

[rju]

JIRA Issue: KIEKER-1283 Split Unit and Integration Tests Original Reporter: Nils Christian Ehmke

---

This has already been discussed multiple times: We should split unit and integration tests. Currently they are mixed all in one directory. Long running "unit" tests could keep developers from actually executing them before a commit. It should be possible to execute unit and integration tests independently in my opinion.

[rju]

author rju -- Sun, 14 Sep 2014 13:52:05 +0200

In testing practice performing integration, performance and other scenario based tests is a totally different task to component tests along well-defined borders. Unit-tests should only focus on in language interfaces. IO can be tested also, but a couple of requirements must be fulfilled:

1. Testing file creation, all files of a test should be located in one common directory for the test, which must be disjunct from all other tests.
2. Other IO tests must also acquire disjunct resources. For TCP, UDP communication, this can be done by allocating different ports for each test. However, there is no system service where you can fetch a free port number for your test. Probing for free ports can be implemented. However, as probing and allocating cannot be performed in one system operation, race conditions may occur. A better solution would the execution of those tests in a virtual machine.

As we do not realize our own IP protocol, we do not need to unit-test that protocol stack level. We can, therefore, test the data before it is send.

For integration tests we could indeed use one or more virtual machines which get their IP address automatically on startup. Such infrastructure could be realized with virtual box or KVM and does not need OpenStack. It might be a good idea to let a bachelor student write his/her thesis on this topic.

In short: I agree with Nils to separate the classical unit tests from the integration tests. As a first step we should separate the tests to an appropriate package without changing its underlying technology.

[rju]

author André van Hoorn -- Thu, 23 Oct 2014 16:41:10 +0200

Covered by KIEKER-1170 Done

[rju]

author Imported User 11 -- Sat, 24 Jan 2015 01:27:45 +0100

U
- Unit Test
I
- Integration Test (or direction of)

Package	Class	U	I	Duration	Port
kieker/test/common
junit	TestJUnitTestsExtendingAbstractKiekerTest		X	0,31 s	-
junit.configuration	TestConfigurationPath	X		0,016 s	-
junit.record	TestRecordSerialization	X		5,521 s	-
	TestRecordsUsingCorrectBinarySize		X	0,35 s	-
junit.record.controlflow	TestOperationExecutionRecordEquals	X		0,016 s	-
	TestOperationExecutionRecordConstructors	X		0,047 s	-
junit.record.factory	CachedRecordFactoryCatalogTest		X	0,031 s	-
	RecordFactoryResolverTest	X		0,016 s	-
junit.record.flow.trace	TestConstructionEvent	X		0,015 s	-
	TestTraceMetadata	X		0,016 s	-
junit.record.flow.trace.concurrency	TestJoinEvent	X		0,027 s	-
	TestSplitEvent	X		0,027 s	-
junit.record.flow.trace.concurrency.monitor	TestMonitorEntryEvent	X		0,026 s	-
	TestMonitorExitEvent	X		0,028 s	-
	TestMonitorWaitEvent	X		0,031 s	-
	TestMonitorRequestEvent	X		0,027 s	-
	TestMonitorNotifyEvent	X		0,032 s	-
	TestMonitorNotifyAllEvent	X		0,034 s	-
junit.record.flow.trace.operation.constructor	TestAfterConstructorEvent	X		0,030 s	-
	TestAfterConstructorFailedEvent	X		0,034 s	-
	TestBeforeConstructorEvent	X		0,028 s	-
	TestCallConstructorEvent	X		0,026 s	-
junit.record.flow.trace.operation.object	TestAfterOperationFailedObjectEvent	X		0,028 s	-
	TestAfterOperationObjectEvent	X		0,033 s	-
	TestBeforeOperationObjectEvent	X		0,029 s	-
	TestCallOperationObjectEvent	X		0,025 s	-
junit.util	TestClassOperationSignaturePair	X		0,017 s	-
junit.util.filesystem	TestFSUtils		X	0,016 s	-
junit.util.map	BoundedConcurrentHashMapTest	X		0,014 s	-
kieker/test/analysis
junit.configuration	OnlineConfigurationTest		X	0,011 s	-
junit.plugin	TestGlobalConfiguration		X	0,042 s	-
	TestPlugin		X	0,061 s	-
	TestPluginConfigurationRetention		X	0,734 s	-
	TestPluginShutdown		X	0,012 s	-
junit.plugin.filter.flow	TestEventRecordTraceReconstructionFilter	X		0,094 s	-
junit.plugin.filter.forward	TestCountingFilter	X		0,051 s	-
	TestStringBufferFilter	X		0,063 s	-
	TestTeeFilter	X		0,141 s	-
junit.plugin.filter.record	TestMonitoringThroughputFilter	X		0,051 s	-
	TestRealtimeRecordDelayFilterAccelerationFaster	X		15,096 s	-
	TestRealtimeRecordDelayFilterAccelerationSlowdown	X		20,099 s	-
	TestRealtimeRecordDelayFilterNoAcceleration	X		20,101 s	-
junit.plugin.filter.select	TestTimestampFilter	X		0,059 s	-
	TypeFilterTest	X		0,041 s	-
junit.plugin.filter.trace	TestTraceIdFilter	X		0,065 s	-
junit.plugin.filter.visualization	TestNoOutputPortsForVisualizations		X	0,349 s	-
junit.plugin.reader	TestNoInputPortsForReader		X	0,348 s	-
junit.plugin.reader.filesystem	TestLegacyExecutionRecordReader	X		0,131 s	-
junit.plugin.reader.namedRecordPipe	TestPipeReader		X	0,055 s	-
junit.plugin.reader.timer	TimeReaderTest	X		28,059 s	-
kieker/test/monitoring
junit.core.configuration	TestConfigurationFactoryMethods	X		0,048 s	-
junit.core.controller	TestAutoSetLoggingTimestamp	X		0,063 s	-
	TestControllerConstruction	X		0,141 s	-
	TestJMXInterface	X		0,015 s	-
	TestMonitoringControllerRecordsPassedInMonitoringStates		X	0,069 s	-
	TestMonitoringControllerStateTransitions	X		0,076 s	-
	TestProbeController		X	10,179 s	-
junit.core.sampler	TestPeriodicSampling	X		16,656 s	-
junit.core.signaturePattern	TestPatternParser	X		3,778 s	-
junit.probe.adaptiveMonitoring.cxf	TestCXFClientServerInterceptors		X	1,653 s	9093
junit.probe.adaptiveMonitoring.mxbean	TestClassLoadingSampler		X	11,5 s	-
	TestCompilationSampler		X	11,5 s	-
	TestGCSampler		X	11,5 s	-
	TestMemorySampler		X	11,5 s	-
	TestThreadsStatusSampler		X	11,5 s	-
	TestUptimeSampler		X	11,5 s	-
junit.probe.adaptiveMonitoring.sigar	TestCPUsCombinedPercSampler		X	8,941 s	-
	TestCPUsDetailedPercSampler		X	8,941 s	-
	TestMemSwapUsageSampler		X	8,916 s	-
junit.probe.adaptiveMonitoring.spring	TestSpringMethodInterceptor		X	5,625 s	9293
junit.probe.cxf.executions	TestCXFClientServerInterceptorsNoSessionRegisteredBefore	X		1,622 s	9091
	TestCXFClientServerInterceptorsSessionRegisteredBefore	X		1,61 s	9092
junit.probe.spring.executions	TestSpringMethodInterceptorInterceptorIsEntryPoint	X		0,157 s	-
	TestSpringMethodInterceptorInterceptorIsNotEntryPoint	X		0,141 s	-
junit.probe.spring.executions.jetty	TestSpringMethodInterceptor	X		3,337 s	-
junit.timer	TestSystemMilliTimer	X		0,047 s	-
	TestSystemNanoTimer	X		0,062 s	-
junit.writer.filesystem	TestLogRotationMaxLogFilesAsyncBinaryFsWriter		X	0,547 s	-
	TestLogRotationMaxLogFilesAyncFsWriter		X	0,485 s	-
	TestLogRotationMaxLogFilesSyncFsWriter		X	0,405 s	-
	TestLogRotationMaxLogSizeAsyncBinaryFsWriter		X	6,229 s	-
	TestLogRotationMaxLogSizeAsyncFsWriter		X	11,845 s	-
	TestLogRotationMaxLogSizeSyncFsWriter		X	11,422 s	-
junit.writer.namedRecordPipe	TestPipeWriter	X		0,078 s	-
kieker/test/tools
junit.bridge	TestJMSClientConnector		X	0,109 s	32446
	TestJMSEmbeddedConnector		X	15,919 s	32447
	TestServiceContainer		X	0,469 s	-
	TestTCPClientConnector		X	0,407 s	32443
	TestTCPMultiServerConnector		X	6,791 s	32444
	TestTCPSingleServerConnector		X	0,862 s	32445
junit.bridge.connector.http	HTTPConnectorTest		X	0,786 s	8585
junit.currentTimeEventGenerator	TestCurrentTimeEventGeneratorFilter	X		0,11 s	-
junit.loggingTimestampConverter	LoggingTimestampConverterToolTest	X		0,094 s	-
junit.logReplayer	TestLogReplayer	X		0,156 s	-
	TestMonitoringRecordLoggerFilter	X		0,234 s	-
junit.opad	OpadIntegrationTest		X	2,08 s	-
junit.opad.filter	AnomalyDetectionFilterTest	X		1,064 s	-
	AnomalyScoreCalculationFilterTest	X		1,065 s	-
	ExtractionFilterTest	X		2,065 s	-
	ForecastingFilterTest	X		2,549 s	-
	RecordConverterTest	X		0,078 s	-
	TimeSeriesPointAggregatorTest	X		0,094 s	-
	UniteMeasurementPairFilterTest	X		0,062 s	-
junit.rbridge	RBridgeTest		X	0,016 s	-
junit.traceAnalysis.filter. executionRecordTransformation	TestExecutionRecordTransformationFilter	X		0,391 s	-
junit.traceAnalysis.filter.flow	TestTraceEventRecords2ExecutionAndMessageTraceFilter	X		0,422 s	-
junit.traceAnalysis.filter. sessionReconstruction	TestSessionReconstructionFilter	X		0,313 s	-
junit.traceAnalysis.filter.traceFilter	TestTraceEquivalenceFilterAssemblyEquivalence	X		0,063 s	-
junit.traceAnalysis.filter.traceReconstruction	TestTraceReconstructionFilter	X		0,093 s	-
junit.traceAnalysis.filter.traceWriter	BasicExecutionTraceWriterFilterTest	X		0,172 s	-
	BasicInvalidExecutionTraceWriterFilterTest	X		0,141 s	-
	BasicMessageTraceWriterFilterTest	X		0,152 s	-
junit.traceAnalysis.filter. visualization.dependencyGraph	ComponentAllocationDependencyGraphTest		X	0,094 s	-
junit.traceAnalysis.filter. visualization.descriptions	DescriptionDecoratorFilterTest		X	0,125 s	-
junit.traceAnalysis.filter. visualization.traceColoring	TraceColoringFilterTest		X	0,11 s	-
junit.traceAnalysis.systemModel	TestExecutionTraceBookstore	X		0,078 s	-
junit.tslib	TimeSeriesPointsBufferTest	X		0,015 s	-
	TimeSeriesTest	X		0,031 s	-
	TimeSeriesTestNoState	X		0,031 s	-
junit.tslib.forecast	ARIMA101ForecasterTest	X		0,031 s	-
	ETSForecasterTest	X		0,031 s	-
	MeanForecasterTest	X		0,031 s	-
	SESRForecasterTest	X		0,031 s	-
junit.writeRead.database	TestAsyncDbWriterReader		X	10,591 s	-
	TestSyncDbWriterReader		X	10,395 s	-
junit.writeRead.filesystem	BasicAsyncAsciiZipWriterReaderTest		X	1,33 s	-
	BasicAsyncBinaryFSWriterReaderTest		X	1,173 s	-
	BasicAsyncBinaryNFSWriterReaderTest		X	1,237 s	-
	BasicAsyncBinaryZipWriterReaderTest		X	1,189 s	-
	BasicAsyncFSWriterReaderTest		X	1,176 s	-
	BasicSyncFSWriterReaderTest		X	1,192 s	-
	DeflateAsyncBinaryFSWriterReaderTest		X	1,158 s	-
	GZipAsyncBinaryFSWriterReaderTest		X	1,158 s	-
	ZipAsyncBinaryFSWriterReaderTest		X	1,17 s	-
junit.writeRead.filesystem.brokenLine	SkipBrokenRecordsRegularFileTest		X	1,205 s	-
junit.writeRead.filesystem.unknownTypes	ContinueAfterUnknownTypeNotForBinaryTest		X	1,19 s	-
	ContinueAfterUnknownTypeRegularFileTest		X	1,158 s	-
	TerminateAfterUnknownTypeNotForBinaryTest		X	1,143 s	-
	TerminateOnFirstUnknownTypeRegularFileTest		X	1,158 s	-
junit.writeRead.jms	BasicJMSWriterReaderTest		X	3,124 s	61616
junit.writeRead.jmx	BasicJMXWriterReaderTest		X	3,25 s	59999
junit.writeRead.namedPipe	BasicNamedRecordPipeWriterReaderTest		X	1,15 s	-
junit.writeRead.printStream	BasicPrintStreamWriterTestFile		X	1,173 s	-
	BasicPrintStreamWriterTestStdErr		X	1,113 s	-
	BasicPrintStreamWriterTestStdout		X	1,094 s	-
junit.writeRead.tcp	BasicTCPWriterReaderTest		X	2,314 s	10333, 10334

[rju]

author nils-christian -- Mon, 23 Feb 2015 16:26:22 +0100

Will be done after Christian refactored the test package structure.

[rju]

author André van Hoorn -- Fri, 8 Sep 2017 16:29:02 +0200

Changed to 1.14.