engine lifecycle / test case isolation

[keturn]

TestInstance.Lifecycle vs MTE Engine lifespan

Currently the Extension does its own management of when to create new Engine instances vs re-using an existing one. (Users choose between MTEExtension or IsolatedMTEExtension.)

JUnit5 introduced its own way to manage test instance lifecycle. A test class may set TestInstance.Lifecycle.PER_CLASS to share state on the instance across invocation of all its test methods.

Is it worthwhile for us to keep our own way to specify whether an engine is shared between tests, or should we drop that part and say the engine's lifespan is the same as the test instance's?

Related Work: There may be other work in progress for managing resources across JUnit tests: https://github.com/junit-pioneer/junit-pioneer/issues/348 started out being about TempDirectory, but may be generalized to support arbitrary types of resources.

Relation of Engine & Game lifecycle to JUnit Before/After hooks

JUnit will call methods annotated with BeforeAll, BeforeEach, AfterAll, AfterEach, as it enters and exits your Test methods.

A clearer way to describe some MTE setup code is according to the game engine's lifecycle: run before or after Engine initialization, or on transition to StateLoading or StateIngame.

GameEngine does provide an interface to subscribe to its state changes, but the interface we have with the most detailed lifecycle methods is the EngineSubsystem.

Should we provide a convenient way to define methods on the test class that are run at these points in the subsystem lifecycle?

What about the ComponentSystem lifecycle methods?

Neither gestalt's Modules nor ModuleEnvironment look to have lifecycle methods, but some things that handle Terasology's ModuleEnvironments call methods on whatever EnvironmentSwitchHandler is in the Context.

[jbduncan]

@keturn I'm the author of the PR to introduce an extension for managing arbitrary resources in junit-pioneer, so thank you for mentioning it!

I can't give a deadline when it will be ready to merge into junit-pioneer, as I still have to iron out some gnarly bugs. But if you still felt up to trying it when it's released, then I'd be honoured. :)

And if you do decide to try out the extension, I'd be even more honoured if you reported any problems or inconveniences you find with it.

[DarkWeird]

@keturn we could reuse the main parts of the engine... if there was no problem here. We have problem with leaking resources between several games(ingame states)

Imho, useful categories of tests(don't mix test categories, please):

1. unit tests : simple one-two classes tests.
2. System-tests: one ComponentSystem tests with mocked other systems ( needs to provide testable eventsystem and componentsystem(lifecycle)
3. EngineSubsystem test: test for one EngineSubsystem, with mocks around and lifecycle
4. Module-wide test: test for one module with testable environment (mock/stub engine's and other module's systems and etc)
5. test whole engine: current MTE.

Common about jupiter Choosing between Per-class or Per-test instances:

• Per-class - you can reuse your resources, your tests is not corrupt data/resources
• Per-test - you cannot reuse your resource, because your test changes data. or you can restore state quickly (e.g. when you changes blocks - you can cleanup with ChunkProvider#purge(Command purgeWorld)

About MTE-related: FYI, Junit have MORE lifecycle points then you describe. We can provide custom callbacks like beforeEngine, beforeStatescribes.

Imho, ideally structure for MTE(per-test):

• MTEExtension#beforeEngine ( you can setup external systems stubs or enginelevel configs there)
• Start engine(state mainmenu-like)
  • beforeAll
  • stateLoading -> stateIngame
    • beforeEach
    • test
    • afterEach
  • stateIngame -> mainmenu-like (success or error)
  • afterAll
• shutdown engine
• MTEExtension#afterEngine

mainmenu-like - state where subsystems and several core systems already ready (modules scanned)`

Another features, which can speed-up runs:

1. disconnect state from engine (can using multiple states from engine... it restrict from state semantics by nice to tests) or using engine pool (reusing several engines during tests)
2. reuse engine between all mte tests.
3. reuse ingamestate between tests, which using same modules in whole workspace (like JS tests from all testclasses and Behaviour tests from all testsclasses can be runs in same stateIngame, bacause this state have this modules) - almost impossible or useless, because tests can be started at omega workspace, single module workspace, CI without Terasology repo...

Features which can helps with testable and parallizing:

1. removing CoreRegistry (MUST DIE!!!) - clearing architecture
2. using gestalt-di (gestalt-di provide more testable systems and another beans - constructor inject from DI best practices)
3. removing EnvironmentSwitchHandler and provide lifecycle mechanism in Managers, Systems, etc.(Maybe callback in state) - clearing architecture (P.S. one of a big mess source in engine. engine module messed with bootstrap code, then SwitchHandler externally cleanuping some code...)

[keturn]

I made an attempt to match the engine instance created by the MTEExtension to the lifecycle of the test instance. I thought that would make things easier!

but it wasn't easier? :confounded: https://github.com/junit-team/junit5/discussions/2918

[keturn]

Fixed by https://github.com/MovingBlocks/Terasology/pull/5039