search

giorgiosironi / phpunit-selenium

Selenium RC integration for PHPUnit
http://www.phpunit.de/
Other
601 stars 270 forks source link

Fail or Skip if selenium server is not running? #450

Open bobemoe opened 3 years ago

bobemoe commented 3 years ago

I know this has been discussed quite a bit, but I'm in exactly the same situation as I know others are; we've just had some bugs get into production because the selenium server failed to start which meant all the phpunit functional tests were skipped and phpunit/CI considered this passed and allowed merge.

I agree that failed is not really appropriate as it hasn't technically failed as the test never ran. But it's also not technically successful, also as the test never ran.

But I also agree at certain points in the testing that all the tests need to run successfully for the CI to be considered passed, and there needs to be some way tell phpunit/selenium that a skipped test is not acceptable at certain stages.

The line in question is Selenium2TestCase.php#L345 and it looks like it might be quite simple to override prepareSession in my tests to change this behaviour.

This is were the functionality was added: https://github.com/giorgiosironi/phpunit-selenium/issues/31 and it was questioned there. And here https://github.com/giorgiosironi/phpunit-selenium/issues/307 but both these issues are very old. What is the current take on this?

Should I be overriding Selenium2TestCase:prepareSession and implementing this myself for each test-suite I maintain?

Should it be implemented within phpunit? They don't think so. https://github.com/sebastianbergmann/phpunit/issues/2840

Should it be implemented in phpunit-selenium??

Or am I really misusing the system?

Thanks all :)

bobemoe commented 3 years ago

OK, so not quite as easy to override Selenium2TestCase:prepareSession as getStrategy() and $session are both private.

Maybe the exception should be allowed to bubble up? This would cause a failed test, but allow the test to catch it and mark as skipped if it so wishes?