These steps actually run in the same PHP process. This is good for debugging/inspection -- but bad for fidelity. In real usage, they run as separate PHP processes.
If you run several tests in the same process, then the relevant files/functions only load once. (The first variant of function xyz() {...} stays in memory.)
This hasn't been a problem before because...
The code-generators were traditionally/typically additive (each test/step makes a new extension/file/function/class).
The code-generators were traditionally/typically passive (they don't necessarily boot CiviCRM; they just fiddle with strings).
But now...
It's starting to lean more on conditional construction (and regeneration) of *.civix.php. Steps aren't just additive -- they can change the content of *.civix.php. (This PR is a good example.)
It's starting to incorporate more active functionality from CiviCRM -- so some steps require booting (while other steps don't). (Increasingly, the files are actually loaded/executed.)
The overall effect is chaotic/painful to trace.
What's probably needed is to change the test-structure to improve isolation. We have a couple high-leverage points:
phpunit's --process-isolation puts each test-case in its own process. But it's not quite realistic -- individual steps would still share the same process.
CivixProjectTestTrait::civix() calls Symfony's CommandTester. Swap that with something similar that uses sub-processes. This could reveal some other hidden interactions, but it's probably better in the long run.
This is an off-shoot of work on #331. Here's the main comment that explains the purpose.
Ah, blerg. There's a test-interaction problem (i.e. tests pass individually but fail in bulk). E2E tests look like this:
function xyz() {...}
stays in memory.)*.civix.php
. Steps aren't just additive -- they can change the content of*.civix.php
. (This PR is a good example.)--process-isolation
puts each test-case in its own process. But it's not quite realistic -- individual steps would still share the same process.CivixProjectTestTrait::civix()
calls Symfony'sCommandTester
. Swap that with something similar that uses sub-processes. This could reveal some other hidden interactions, but it's probably better in the long run.So this PR implements the second approach.