search

symfony / panther

A browser testing and web crawling library for PHP and Symfony
MIT License
2.95k stars 224 forks source link

Browser closes on interactive mode after test fail #427

Open shadowc opened 3 years ago

shadowc commented 3 years ago

Hey, either I'm doing something wrong or there's something going on with interactive mode.

phpunit.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!-- https://phpunit.readthedocs.io/en/latest/configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd" backupGlobals="false" colors="true" bootstrap="tests/bootstrap.php">
  <coverage processUncoveredFiles="true">
    <include>
      <directory suffix=".php">src</directory>
    </include>
  </coverage>

  <php>
    <ini name="error_reporting" value="-1"/>
    <server name="APP_ENV" value="test" force="true"/>
    <server name="SHELL_VERBOSITY" value="-1"/>
    <server name="SYMFONY_PHPUNIT_REMOVE" value=""/>
    <server name="SYMFONY_PHPUNIT_VERSION" value="9.5.1"/>
  </php>

  <testsuites>
    <testsuite name="Project Test Suite">
      <directory>tests</directory>
    </testsuite>
  </testsuites>

  <listeners>
    <listener class="Symfony\Bridge\PhpUnit\SymfonyTestsListener"/>
  </listeners>

  <extensions>
    <extension class="DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension"/>
    <extension class="Symfony\Component\Panther\ServerExtension"/>
  </extensions>
</phpunit>

command:

PANTHER_NO_HEADLESS=1 bin/phpunit --debug

console output:

ime: 00:09.015, Memory: 50.50 MB

There was 1 failure:

1) App\Tests\e2e\AdminBasicTest::testBasicAdminPageLoads
Fail to find the logout link

/home/shadowc/web-local/decoramemas/tests/e2e/AdminBasicTest.php:37

FAILURES!
Tests: 142, Assertions: 622, Failures: 1.

Remaining indirect deprecation notices (1)

  1x: The "DAMA\DoctrineTestBundle\Doctrine\DBAL\AbstractStaticDriverV2" class implements "Doctrine\DBAL\Driver\ExceptionConverterDriver" that is deprecated.
    1x in PHPUnitExtension::executeBeforeFirstTest from DAMA\DoctrineTestBundle\PHPUnit

Other deprecation notices (148)

  148x: Since symfony/monolog-bridge 5.2: Passing an actionLevel (int|string) as constructor's 3rd argument of "Symfony\Bridge\Monolog\Handler\FingersCrossed\HttpCodeActivationStrategy" is deprecated, "Monolog\Handler\FingersCrossed\ActivationStrategyInterface" expected.
    23x in UserManagerTest::setUp from App\Tests\Services
    22x in SuppliersTest::setUp from App\Tests\Controller\Admin\API
    21x in GoodsCategoriesTest::setUp from App\Tests\Controller\Admin\API
    14x in UserEntityTest::setUp from App\Tests\Entity
    8x in RemoveUserRoleCommandTest::setUp from App\Tests\Command
    8x in AddUserRoleCommandTest::setUp from App\Tests\Command
    7x in AdminControllerTest::setUp from App\Tests\Controller\Admin
    7x in AddUserCommandTest::setUp from App\Tests\Command
    6x in SupplierEntityTest::setUp from App\Tests\Entity
    6x in GoodsCategoryEntityTest::setUp from App\Tests\Entity
    5x in SetUserPasswordCommandTest::setUp from App\Tests\Command
    4x in RemoveUserCommandTest::setUp from App\Tests\Command
    3x in SuppliersTest::testThatWeShouldBeLoggedIn from App\Tests\Controller\Admin\API
    3x in GoodsCategoriesTest::testThatWeShouldBeLoggedIn from App\Tests\Controller\Admin\API
    1x in ListUserRolesCommandTest::testAddUserWithNoArguments from App\Tests\Command
    1x in LoginControllerTest::testLoginFormBasics from App\Tests\Controller\Admin
    1x in LoginControllerTest::testLoginFormStructure from App\Tests\Controller\Admin
    1x in ListUserRolesCommandTest::testListRolesNonexistentUser from App\Tests\Command
    1x in LogoutControllerTest::testWeCanLogout from App\Tests\Controller\Admin
    1x in BasicURICheckoutTest::testHomePageLoads from App\Tests\Controller
    1x in BasicURICheckoutTest::testHttpRedirects from App\Tests\Controller
    1x in BasicURICheckoutTest::testBasicURICheckout from App\Tests\Controller
    1x in ListUserRolesCommandTest::testWeCanListAUserRoles from App\Tests\Command
    1x in ListUserRolesCommandTest::testCommandExists from App\Tests\Command
    1x in LogoutControllerTest::setUp from App\Tests\Controller\Admin

Right after the test fails, the browser closes. So I can't find out what is the reason for the failed test.

OskarStark commented 3 years ago

Could you add $client->takeScreenshot($pathToFile) before/after sth. to gather more information?

shadowc commented 3 years ago

Yes, absolutely! Will do tomorrow or later today after valentine's dinner :p!

shadowc commented 3 years ago

@OskarStark Can you be a bit more specific on what you mean by before/after sth.?

If I do a takeScreenshot after the test fails, the screenshot doesn't get taken. If I take the screenshot right when the test is about to fail, I get a normal browser screenshot (pretty much what I'm looking at when I'm waiting for the test to fail).

The browser window then gets closed.

screenshot

I now have a better idea of what's happening though: I'm clicking the login button, and then immediately switching pages (not waiting for the http request to finish!). In this regard everything works as expected with the panther driver itself. It's just that I don't get a "pause" and I can't look around the browser window/console when this happens.

shadowc commented 3 years ago

@OskarStark ping!

I'm pasting the test code in case this also helps!

<?php

namespace App\Tests\e2e;

use Exception;
use Symfony\Component\Panther\Client;
use Symfony\Component\Panther\PantherTestCase;

class AdminBasicTest extends PantherTestCase
{
    private bool $initialized = false;
    private Client $client;

    public function setUp(): void
    {
        if (!$this->initialized) {
            $this->client = static::createPantherClient(
                ['external_base_uri' => 'https://localhost:8000'],
                ['HTTPS' => true],
                ['capabilities' => [
                    'acceptInsecureCerts' => true,
                ]],
            );
            $this->initialized = true;
        }

        $this->doLogin();
    }

    public function testBasicAdminPageLoads()
    {
        try {
            $this->client->waitFor('div > ul > li > a[href="/logout"]', 5);
        } catch (Exception $exception) {
            $this->fail('Fail to find the logout link');
        }

        $this->client->click(
            $this->client->getCrawler()->filter('div > ul > li > a[href="/logout"]')->first()->link()
        );

        try {
            $this->client->waitFor('div#container h1.hidden', 5);
        } catch (Exception $exception) {
            $this->fail('Failed to reach the main page');
        }

        $this->assertStringContainsString(
            'localhost:8000/',
            $this->client->getCurrentURL(),
            'Test we can logoff admin'
        );

        $this->client->request('GET', '/admin');

        try {
            $this->client->waitFor('input#username', 5);
        } catch (Exception $exception) {
            $this->fail('Failed to go back to login page');
        }

        $this->assertStringContainsString(
            '/login',
            $this->client->getCurrentURL(),
            'Test we can logoff admin'
        );
    }

    private function doLogin(): void
    {
        $crawler = $this->client->request('GET', '/login');

        try {
            $this->client->waitFor('input#username', 5);
        } catch (Exception $exception) {
            $this->fail('Failed to find the login form');
        }

        $form = $crawler->selectButton('Ingresar')->form();
        $form->setValues([
            'username' => 'shadowc',
            'password' => '123',
        ]);

        $this->client->submit($form);
    }
}

At this point the test fails on interactive mode because the browser window is not big enough to make the "logout" link "interactable". It pass in non-interactive mode. (this issue is being discussed in another github issue)

Regardless of the reasons (in before, this tests would fail for a different reason that I've stated above), the browser window closes and we don't get a pause (hence, we cannot debug failed tests).

I'd LOVE for this to be fixed (alas, I have little knowledge of browser drivers to help, but may be there's still something I can do?) so that I can migrate all my cypress tests to panther.

Other features needed IMO are:

shadowc commented 3 years ago

Another update: I made the test fail and run it with: 

PANTHER_ERROR_SCREENSHOT_DIR="./" PANTHER_NO_HEADLESS=1 bin/phpunit --debug ./tests/e2e

This worked! But it still failed to pause the browser (chrome closes right after the test fails). Seems almost like the extension is not correctly hooked into phpunit.

Here's my composer.json in case this brings some light:

{
    "type": "project",
    "license": "proprietary",
    "require": {
        "php": "^7.4.12",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "ext-json": "*",
        "composer/package-versions-deprecated": "^1.11",
        "doctrine/doctrine-bundle": "^2.2",
        "doctrine/doctrine-fixtures-bundle": "^3.4",
        "doctrine/doctrine-migrations-bundle": "^3.0",
        "doctrine/orm": "^2.7",
        "friendsofphp/php-cs-fixer": "^2.16",
        "gedmo/doctrine-extensions": "^3.0",
        "google/recaptcha": "^1.2",
        "knplabs/knp-markdown-bundle": "^1.8",
        "sentry/sentry-symfony": "^3.5",
        "symfony/asset": "5.2.*",
        "symfony/console": "5.2.*",
        "symfony/debug-bundle": "5.2.*",
        "symfony/dotenv": "5.2.*",
        "symfony/flex": "^1.11.0",
        "symfony/framework-bundle": "5.2.*",
        "symfony/monolog-bundle": "^3.6",
        "symfony/security-bundle": "5.2.*",
        "symfony/swiftmailer-bundle": "^3.4",
        "symfony/twig-bundle": "5.2.*",
        "symfony/var-dumper": "5.2.*",
        "symfony/web-profiler-bundle": "5.2.*",
        "symfony/yaml": "5.2.*",
        "twig/extra-bundle": "^3.1",
        "twig/twig": "^3.1"
    },
    "require-dev": {
        "dama/doctrine-test-bundle": "^6.5",
        "dbrekelmans/bdi": "^0.3.0",
        "symfony/browser-kit": "5.2.*",
        "symfony/css-selector": "5.2.*",
        "symfony/panther": "^1.0",
        "symfony/phpunit-bridge": "^5.2"
    },
    "config": {
        "preferred-install": {
            "*": "dist"
        },
        "sort-packages": true
    },
    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "App\\Tests\\": "tests/"
        }
    },
    "replace": {
        "paragonie/random_compat": "2.*",
        "symfony/polyfill-ctype": "*",
        "symfony/polyfill-iconv": "*",
        "symfony/polyfill-php72": "*",
        "symfony/polyfill-php71": "*",
        "symfony/polyfill-php70": "*",
        "symfony/polyfill-php56": "*"
    },
    "scripts": {
        "auto-scripts": {
            "cache:clear": "symfony-cmd",
            "assets:install %PUBLIC_DIR%": "symfony-cmd"
        },
        "post-install-cmd": [
            "@auto-scripts"
        ],
        "post-update-cmd": [
            "@auto-scripts"
        ]
    },
    "conflict": {
        "symfony/symfony": "*"
    },
    "extra": {
        "symfony": {
            "allow-contrib": false,
            "require": "5.2.*"
        }
    }
}
shadowc commented 3 years ago

Ping!

I've forked the panther repo and all its tests pass in my dev pc (Ubuntu 20.04). Do we have coverage tests for interactive clients?

If not, I'm not 100% sure how to write a test for that, but if you can may be point me in the right direction, I could start investigating and debugging possible reasons why this fails, and try to come up with a solution!

shadowc commented 3 years ago

Ping!

I've found an issue with this issue :p

I've found the test that is supposed to check weather the browser will be paused on test failure. The problem with the test itself is that it skips calling $this->pause(sprintf('Failure: %s', $message));:

public function testPauseOnFailure(string $method, string $expected): void
{
    $extension = new ServerExtension();
    $extension->testing = true;
    ...

In the beginning of the test, this flag is set. Later, in pause() we see this:

if (!$this->testing) {
    fgets(\STDIN);
}

And this is why a test wouldn't cover the issue reported here. So I added some echo lines throughout the code to debug a little bit and I've found out that in my particular case, when I reach the test failure, PantherTestCase::isWebServerStarted() is false, so the pause is never done:

(with my debug lines)

private function pause($message): void
{
    echo PantherTestCase::isWebServerStarted() ? "Web server started\n" : "Web server hasn't started!!\n";

    if (PantherTestCase::isWebServerStarted()
        && \in_array('--debug', $_SERVER['argv'], true)
        && $_SERVER['PANTHER_NO_HEADLESS'] ?? false
    ) {
        echo "$message\n\nPress enter to continue...";
        if (!$this->testing) {
            fgets(\STDIN);
        }
    }
}

And the output is:

Testing /home/shadowc/web-local/decoramemas/tests/e2e
Test 'App\Tests\e2e\AdminBasicTest::testBasicAdminPageLoads' started
Web server hasn't started!!
Test 'App\Tests\e2e\AdminBasicTest::testBasicAdminPageLoads' ended

Thus the pause isn't performed. Am I correct to assume that the browser for some reason has closed and therefore, pausing is irrelevant at this point?

shadowc commented 3 years ago

Update: not at all. Removing that first check, does make the browser window stick open:

private function pause($message): void
{
    echo PantherTestCase::isWebServerStarted() ? "Web server started\n" : "Web server hasn't started!!\n";

    if (//PantherTestCase::isWebServerStarted()
        /*&&*/ \in_array('--debug', $_SERVER['argv'], true)
        && $_SERVER['PANTHER_NO_HEADLESS'] ?? false
    ) {
        echo "$message\n\nPress enter to continue...";
        if (!$this->testing) {
            fgets(\STDIN);
        }
    }
}