search

lstrojny / phpunit-clever-and-smart

Smarter test runner for PHPUnit
170 stars 13 forks source link

Skipped test classes crash phpunit #38

Open willemstuursma opened 8 years ago

willemstuursma commented 8 years ago

The following test will crash PHPUnit if phpunit-clever-and-smart is registered as a test listener:

/**
 * Class ExampleCrashTest
 *
 * @requires function i_dont_exist
 */
class ExampleCrashTest extends PHPUnit_Framework_TestCase
{

}

Apparently, if an entire class is skipped instead of a single test method, a PHPUnit_Framework_TestSuite object is passed to \PHPUnit\Runner\CleverAndSmart\TestListener::addSkippedTest. This object implements the \PHPUnit_Framework_Test interface. However in \PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage::record, suddenly a \PHPUnit_Framework_TestCase object is required. The typehint fails, crashing the phpunit process.

Fatal error: Uncaught exception 'ErrorException' with message 'Argument 2 passed to PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage::record() must be an instance of PHPUnit_Framework_TestCase, instance of PHPUnit_Framework_TestSuite given, called in /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php on line 107 and defined' in /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php on line 73

Call stack:

Call Stack:
    0.0003     225232   1. {main}() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/phpunit:0
    0.0099     712344   2. PHPUnit_TextUI_Command::main() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/phpunit:47
    0.0099     712984   3. PHPUnit_TextUI_Command->run() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:109
    0.0099     715912   4. PHPUnit_TextUI_Command->handleArguments() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:120
    0.0160    1146544   5. PHPUnit_TextUI_Command->handleBootstrap() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:633
    0.0166    1154992   6. PHPUnit_Util_Fileloader::checkAndLoad() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:804
    0.0168    1155152   7. PHPUnit_Util_Fileloader::load() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Util/Fileloader.php:38
    0.0170    1167880   8. include_once('/Users/willem/Sites/mollie/tests/bootstrap.php') /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Util/Fileloader.php:56
    0.6078    5443552   9. require_once('/Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php') /Users/willem/Sites/mollie/tests/bootstrap.php:47
    0.6102    5496072  10. IDE_Base_PHPUnit_TextUI_Command::main() /Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php:553
    0.6102    5496696  11. PHPUnit_TextUI_Command->run() /Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php:296
    0.6202    6172160  12. PHPUnit_TextUI_TestRunner->doRun() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:158
    0.6437    6470952  13. PHPUnit_Framework_TestSuite->run() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:436
    0.6551    6810536  14. PHPUnit_Framework_TestResult->addFailure() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Framework/TestSuite.php:705
    0.6740    8401768  15. PHPUnit\Runner\CleverAndSmart\TestListener->addSkippedTest() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Framework/TestResult.php:329
    0.6741    8401848  16. PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->record() /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php:107
    0.6741    8403248  17. PHPUnit_Util_Fileloader::{closure:/Users/willem/Sites/mollie/core/tests/bootstrap.php:15-23}() /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php:73
epdenouden commented 6 years ago

@willemstuursma thanks for this issue, it makes a good edge case for testing rebustness of test execution reordering. I've have added it as a test for my Breakfast project: https://github.com/epdenouden/phpunit/blob/breakfast/tests/Runner/TestSuiteSorterEmptyTestCaseTest.php Let me know if you have any questions or requests!