Unable to run tests for the project

berliner commented 7 years ago

In order to test some changes that I'd like to issue a PR for, I tried to run the tests for MinkZombieDriver, but I can't make them pass on a clean repo checkout.

$ which node
/opt/local/bin/node

$ node -v
v4.6.1

$ which npm
/opt/local/bin/npm

$ npm -v
3.10.10

$ git clone https://github.com/minkphp/MinkZombieDriver.git
$ cd MinkZombieDriver/
$ sudo npm -g install zombie
$ composer install

$ phpunit
PHPUnit 4.5.0 by Sebastian Bergmann and contributors.

Configuration read from /private/tmp/mink/MinkZombieDriver/phpunit.xml.dist

........................EEEE..........EEEEEEEEEEEEEEEEEEEEEEEEE  63 / 190 ( 33%)
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEESSSSEEEEEEEEEEEEEEEESSSSE 126 / 190 ( 66%)
EEEEEEEEEEEEEEESEEEEEEEEEEEEEEEEESEEEEEEEESEEEEEEEEEEEEEEEEEEEE 189 / 190 ( 99%)
E

Time: 10.48 seconds, Memory: 14.75Mb

There were 145 errors:

1) Behat\Mink\Tests\Driver\Basic\BasicAuthTest::testSetBasicAuth with data set #0 ('mink-user', 'mink-password', 'is authenticated')
RuntimeException: No connection available. Did you start the server?

/private/tmp/mink/MinkZombieDriver/src/NodeJS/Server.php:441
/private/tmp/mink/MinkZombieDriver/src/NodeJS/Server.php:411
/private/tmp/mink/MinkZombieDriver/src/ZombieDriver.php:197
/private/tmp/mink/MinkZombieDriver/vendor/behat/mink/src/Session.php:159
/private/tmp/mink/MinkZombieDriver/vendor/mink/driver-testsuite/tests/Basic/BasicAuthTest.php:16

...

Any ideas what I might do wrong? I'm on OS X 10.12.5 if that matters.

aik099 commented 7 years ago

Can you please:

upgrade your node/npm
show version of Zombie being used ?

stof commented 7 years ago

you need to run the fixtures server:

vendor/bin/mink-test-server if you want to block a terminal with it (allowing to stop it with Ctrl+C
vendor/bin/mink-test-server > /dev/null 2>&1 & to run it silently in the background (you then have to find the process id and kill it to stop it)

berliner commented 7 years ago

$ git clone https://github.com/minkphp/MinkZombieDriver.git
$ cd MinkZombieDriver/
$ npm install zombie
npm WARN saveError ENOENT: no such file or directory, open '/private/tmp/mink/MinkZombieDriver/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/private/tmp/mink/MinkZombieDriver/package.json'
npm WARN MinkZombieDriver No description
npm WARN MinkZombieDriver No repository field.
npm WARN MinkZombieDriver No README data
npm WARN MinkZombieDriver No license field.

+ zombie@5.0.5
added 98 packages in 9.699s

$ composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 5 installs, 0 updates, 0 removals
  - Installing symfony/phpunit-bridge (v3.3.2): Loading from cache
  - Installing symfony/css-selector (v3.3.2): Loading from cache
  - Installing behat/mink (dev-master 9ea1ceb): Cloning 9ea1cebe3d from cache
  - Installing mink/driver-testsuite (dev-master 1d2e9fc): Cloning 1d2e9fcef1 from cache
  - Installing symfony/process (v3.3.2): Loading from cache
symfony/phpunit-bridge suggests installing symfony/debug (For tracking deprecated interfaces usages at runtime with DebugClassLoader)
symfony/phpunit-bridge suggests installing ext-zip (Zip support is required when using bin/simple-phpunit)
behat/mink suggests installing behat/mink-browserkit-driver (extremely fast headless driver for Symfony\Kernel-based apps (Sf2, Silex))
behat/mink suggests installing behat/mink-goutte-driver (fast headless driver for any app without JS emulation)
behat/mink suggests installing behat/mink-selenium2-driver (slow, but JS-enabled driver for any app (requires Selenium2))
Writing lock file
Generating autoload files

$ node -v
v8.1.0

$ npm -v
5.0.3

(In separate terminal window, to make the server work on my sytem, I had to replace readlink with greadlink)
$ vendor/bin/mink-test-server
PHP 5.6.27 Development Server started at Tue Jun 13 15:22:17 2017
Listening on http://localhost:8002
Document root is /private/tmp/mink/MinkZombieDriver/vendor/mink/driver-testsuite/web-fixtures
Press Ctrl-C to quit.

(In the original terminal window)
$ phpunit --stop-on-failure
PHPUnit 4.5.0 by Sebastian Bergmann and contributors.

Configuration read from /private/tmp/mink/MinkZombieDriver/phpunit.xml.dist

........................E

Time: 152 ms, Memory: 9.25Mb

There was 1 error:

1) Behat\Mink\Tests\Driver\Basic\BasicAuthTest::testSetBasicAuth with data set #0 ('mink-user', 'mink-password', 'is authenticated')
RuntimeException: No connection available. Did you start the server?

/private/tmp/mink/MinkZombieDriver/src/NodeJS/Server.php:441
/private/tmp/mink/MinkZombieDriver/src/NodeJS/Server.php:411
/private/tmp/mink/MinkZombieDriver/src/ZombieDriver.php:197
/private/tmp/mink/MinkZombieDriver/vendor/behat/mink/src/Session.php:159
/private/tmp/mink/MinkZombieDriver/vendor/mink/driver-testsuite/tests/Basic/BasicAuthTest.php:16

FAILURES!
Tests: 25, Assertions: 62, Errors: 1.

Legacy deprecation notices (2)

stof commented 7 years ago

hmm, the driver is not tested with Zombie 5 currently, so it may be broken (I was not aware that they released a version 5). Try with version 4 (use npm install zombie@^4 for the install command).

regarding readlink vs greadlink, this is possible indeed. I actually never tried the server script on Mac OS X (as I don't have such environment). The script was used on Linux for now (I am on Linux, and our CI is as well)

berliner commented 7 years ago

I just tried with zombiejs 4.3.0 with the same result.

I have also tried to follow the steps above on a pretty clean apricity linux, using the same node, npm and phpunit versions, the result is the same.

aik099 commented 7 years ago

Actually I've tried to run it myself now (Zombie 4.2.1, Node 5.4.1, Npm 3.3.12) and it failed with same errors as you've seen. :(

berliner commented 7 years ago

@aik099 That's good news to me, so it's not only me ;)

aik099 commented 7 years ago

Apparently Zombie did change something again to break everything for us.

berliner commented 7 years ago

Do the test on your Travis CI still work?

aik099 commented 7 years ago

I've restarted a couple of jobs from the latest build (1st and last): https://www.travis-ci.org/minkphp/MinkZombieDriver/builds/175044232 . Let's see if they will pass.

aik099 commented 7 years ago

It did fail in some places, but not that massively as it did when we've executed it locally:

Zombie 4.3.0: https://www.travis-ci.org/minkphp/MinkZombieDriver/jobs/175044245
Zombie 3.1.1: https://www.travis-ci.org/minkphp/MinkZombieDriver/jobs/175044233

berliner commented 7 years ago

@aik099 Will you be working on this? I fear that my knowledge of this project is not sufficient to effectively address these problems.

aik099 commented 7 years ago

Unfortunately I'm occupied with other things at the moment.

Here is approach I've used to solve Zombie issues:

determine Zombie version for which all tests passed (you can get it from last passed build on Travis CI)
try increasing used Zombie version until problem starts to appear
see what changes were made to Zombie that might have caused issues with tests
if it isn't Zombie bug try adopting driver code to work with older and newer version of Zombie
repeat process until all tests pass for latest Zombie 5 version

berliner commented 7 years ago

Ok, after further digging I haven't found a problem with zombie itself, but instead I have found these 2 problems in MinkZombieDriver and Mink:

Session start: https://github.com/minkphp/Mink/commit/9ea1cebe3dc529ba3861d87c818f045362c40484 introduced lazy starting of sessions when calling the visit method. This makes all the tests in Behat\Mink\Tests\Driver\Basic\BasicAuthTest create errors. If I add the following code to the top of Behat\Mink\Session::setBasicAuth the the tests pass:
```
    // start session if needed
    if (!$this->isStarted()) {
        $this->start();
    }
```
HTML5 tests: https://github.com/minkphp/driver-testsuite/pull/6/commits/55ea007c11bc67b097eb0c06e65f5c8c6fe44df0 added 2 new tests for HTML5 form elements. These are not yet fully supported by zombiejs and should therefor be added to the exclude list in Behat\Mink\Tests\Driver\ZombieConfig::skipMessage

aik099 commented 7 years ago

1 - that is unexpected considering that according to documentation (see https://github.com/minkphp/Mink/blob/master/src/Session.php#L57-L67) the setBasicAuth method should work prior to session being started

2 - sure, please send a PR fixing that

FYI: The support for setting authentication headers is currently limited to these drivers:

ZombieDriver basicAuth: https://github.com/minkphp/MinkZombieDriver/blob/master/src/ZombieDriver.php#L168
BrowserKit basicAuth: https://github.com/minkphp/MinkBrowserKitDriver/blob/master/src/BrowserKitDriver.php#L192

The BrowserKit is setting headers, that would be picked up upon session start, but the Zombie apparently wants to start session first and then set authentication to be picked up by next visit call.

aik099 commented 7 years ago

1 - since apparently Zombie expects session to be started prior to setting auth info we need to do this on driver level instead of session level.

@stof , thoughts?

berliner commented 7 years ago

@aik099 For 2. Sure, I created https://github.com/minkphp/MinkZombieDriver/pull/180

stof commented 7 years ago

@aik099 yeah, it would make sense to start the session at the driver level in this case (the starting actually happens at the driver anyway. Starting a session just delegates starting to the driver)

aik099 commented 7 years ago

@berliner , does it work for you now, when PR was merged?

berliner commented 7 years ago

@aik099 Yes, the test are passing now:

$ phpunit
PHPUnit 4.5.0 by Sebastian Bergmann and contributors.

Configuration read from /private/tmp/mink/MinkZombieDriver/phpunit.xml.dist

...............................................................  63 / 190 ( 33%)
..........................S...S......SSSSS...............SSSS.S 126 / 190 ( 66%)
S.............S..........................S...............S..SSS 189 / 190 ( 99%)

Time: 15.9 seconds, Memory: 11.75Mb

OK, but incomplete, skipped, or risky tests!
Tests: 187, Assertions: 438, Skipped: 19.

Legacy deprecation notices (2)

aik099 commented 7 years ago

Then you can safely close the issue I guess.

berliner commented 7 years ago

Ah, right. Thanks for your support.

minkphp / MinkZombieDriver

Unable to run tests for the project #179