Currently the test runner uses a PHPUnit PHAR for executing the tests and then processes the produced log files. This is an approach using many moving parts:
PHPUnit PHAR
2 log files in temporary directory
composer project in junit-handler
contains PHPUnit as source package
2 log file parsers
1 output file producer
and surrounding Docker builder, tests etc.
Reasoning
I do not have any information about the rationals behind these choices.
Suggested approach
My suggestion reduces these moving parts and the amount of code required:
composer project on main level
contains PHPUnit as source package (no PHPUnit PHAR)
Implement a PHPUnit logger producing exercism JSON result directly
1 output (file) producer
and surrounding Docker builder, tests etc.
Reasoning
PHPUnit has done a lot of work to provide a modern, robust, documented event system for custom extensions like loggers. The PHPUnit developers also promised to keep this system as stable as possible in the future.
The events contain all information that is also rendered into the loggers bundled with PHPUnit. By using the event information directly, a wider range of information can be forwarded to the students (e.g. PHP warnings, deprecations etc.) without using commandline switches and log file parsing. This reduces the code required to get the information in the first place.
PHPUnit also offers a way to introduce custom commandline parameters. This can be used to parameterize the output location of the log file, like it is done now for JUnit / Teamcity logs. Another possible way to solve this could be using environment variables (e.g. with the exercise slug). Or we may even replace PHPUnit output with the exercism JSON (writing it to stdout) and capture the output into the required file on shell level.
We may by this reduce the effort to maintain code between PHPUnit versions (events are most likely added, not removed / changed) and the size of the Docker image (probably not worth mentioning).
There was another approach suggested in #75. This suggestion contained some breaking changes, like adding attributes for task ids (which would not exist for students using the CLI) or replacing the PHPUnit commandline interface with an own symfony/console (requiring redesigning the shell scripts). While there is an overlapping set of possibilities, the suggested approach does not aim at more than reducing the moving parts for the inner working of PHPUnit + JSON result production.
Current approach
Currently the test runner uses a PHPUnit PHAR for executing the tests and then processes the produced log files. This is an approach using many moving parts:
Reasoning
I do not have any information about the rationals behind these choices.
Suggested approach
My suggestion reduces these moving parts and the amount of code required:
Reasoning
PHPUnit has done a lot of work to provide a modern, robust, documented event system for custom extensions like loggers. The PHPUnit developers also promised to keep this system as stable as possible in the future.
The events contain all information that is also rendered into the loggers bundled with PHPUnit. By using the event information directly, a wider range of information can be forwarded to the students (e.g. PHP warnings, deprecations etc.) without using commandline switches and log file parsing. This reduces the code required to get the information in the first place.
PHPUnit also offers a way to introduce custom commandline parameters. This can be used to parameterize the output location of the log file, like it is done now for JUnit / Teamcity logs. Another possible way to solve this could be using environment variables (e.g. with the exercise slug). Or we may even replace PHPUnit output with the exercism JSON (writing it to
stdout) and capture the output into the required file on shell level.We may by this reduce the effort to maintain code between PHPUnit versions (events are most likely added, not removed / changed) and the size of the Docker image (probably not worth mentioning).
There was another approach suggested in #75. This suggestion contained some breaking changes, like adding attributes for task ids (which would not exist for students using the CLI) or replacing the PHPUnit commandline interface with an own
symfony/console(requiring redesigning the shell scripts). While there is an overlapping set of possibilities, the suggested approach does not aim at more than reducing the moving parts for the inner working of PHPUnit + JSON result production.