search

consolidation / robo

Modern task runner for PHP
http://robo.li
Other
2.66k stars 305 forks source link

Codeception tests ran with Robo Parallel has errors 0 and timeout 0.000000 in single test but in main raport all tests has errors #939

Open michalper opened 4 years ago

michalper commented 4 years ago

What are you trying to achieve?

I trying to run acceptance tests on TeamCity using Robo Parallel.

What do you get instead?

Every test has assertions="0" time="0.000000" in XML report, but in main report there is information:

<testsuite name="acceptance" tests="12" assertions="0" errors="12" failures="0" skipped="0" time="0.000000">

Provide console output if related. Use -vvv mode for more details.

  Starting: /usr/bin/php vendor/bin/robo -vvv parallel:run
07:00:34
  in directory: /opt/teamcity/buildAgent/work/28e4ceac98318065
07:00:34
   [ParallelExec] /opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_1 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_1.xml
07:00:34
   [ParallelExec] /opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_2 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_2.xml
07:00:34
   [ParallelExec] /opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_3 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_3.xml
07:00:34
   [ParallelExec] /opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_4 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_4.xml
07:00:34
   [ParallelExec] /opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_5 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_5.xml
07:00:43
   [ParallelExec]  '/opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_1 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_1.xml' exited with code 255 
07:00:43
  '/opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_2 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_2.xml' exited with code 1 
07:00:43
  '/opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_3 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_3.xml' exited with code 1 
07:00:43
  '/opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_4 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_4.xml' exited with code 1 
07:00:43
  '/opt/teamcity/buildAgent/work/28e4ceac98318065/vendor/bin/codecept run acceptance --group paracept_5 --debug -c /opt/teamcity/buildAgent/work/28e4ceac98318065/codeception.yml --xml /opt/teamcity/buildAgent/work/28e4ceac98318065/tests/_output/result_5.xml' exited with code 1 
07:00:43
    Time 8.171s
07:00:43
   [ParallelExec]  Exit code 255  Time 8.171s
07:00:43
  Process exited with code 255
07:00:43
  Process exited with code 255 (Step: Acceptance tests run parallel (Command Line))
<testsuites>
<testsuite name="acceptance" tests="12" assertions="0" errors="12" failures="0" skipped="0" time="0.000000">
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesCreateCest.php" name="createClientWebsite" class="Tests\Acceptance\ClientsWebsites\ClientsWebsitesCreateCest" feature="create client website" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesDeleteCest.php" name="deleteClientWebsite" class="Tests\Acceptance\Clients\ClientsWebsitesDeleteCest" feature="delete client website" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesSearchCest.php" name="searchUsingClientId" class="Tests\Acceptance\ClientsWebsites\ClientsWebsitesSearchCest" feature="Search using client_id select" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesSearchCest.php" name="searchUsingProductId" class="Tests\Acceptance\ClientsWebsites\ClientsWebsitesSearchCest" feature="Search using product_id select" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesSearchCest.php" name="searchUsingWebsiteUrl" class="Tests\Acceptance\ClientsWebsites\ClientsWebsitesSearchCest" feature="Search using website_url input" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/ClientsWebsites/ClientsWebsitesSearchCest.php" name="searchUsingExpirationDate" class="Tests\Acceptance\ClientsWebsites\ClientsWebsitesSearchCest" feature="Search using website_url input" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesCreateCest.php" name="createLanguage" class="Tests\Acceptance\Countries\CountriesCreateCest" feature="create language" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesDeleteCest.php" name="deleteCountry" class="Tests\Acceptance\Countries\CountriesDeleteCest" feature="delete country" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesEditCest.php" name="editCountry" class="Tests\Acceptance\Countries\CountriesEditCest" feature="edit country" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesSearchCest.php" name="seeLanguages" class="Tests\Acceptance\Countries\CountriesSearchCest" feature="See list of countries" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesSearchCest.php" name="searchUsingName" class="Tests\Acceptance\Countries\CountriesSearchCest" feature="Search using name input" assertions="0" time="0.000000"/>
<testcase file="/opt/teamcity/buildAgent/work/28e4ceac98318065/tests/acceptance/Countries/CountriesSearchCest.php" name="searchUsingContinent" class="Tests\Acceptance\Countries\CountriesSearchCest" feature="Search using continent_id select" assertions="0" time="0.000000"/>
</testsuite>
</testsuites>

Provide test source code if related

<?php

require_once 'vendor/autoload.php';

use Codeception\Task\MergeReports;
use Codeception\Task\SplitTestsByGroups;
use Robo\Result;
use Robo\Tasks;

class RoboFile extends Tasks
{
    use MergeReports;
    use SplitTestsByGroups;

    public function parallelSplitTests(): void
    {
        $this->taskSplitTestFilesByGroups(5)
            ->projectRoot('.')
            ->testsFrom('tests/acceptance')
            ->groupsTo('tests/_data/paracept_')
            ->run();
    }

    public function parallelRun(): Result
    {
        $parallel = $this->taskParallelExec();
        for ($i = 1; $i <= 5; $i++) {
            $parallel->process(
                $this->taskCodecept()
                    ->suite('acceptance')
                    ->group("paracept_$i")
                    ->debug()
                    ->configFile(__DIR__ . '/codeception.yml')
                    ->xml(__DIR__ . "/tests/_output/result_$i.xml")
            );
        }

        return $parallel->run();
    }

    public function parallelMergeResults(): void
    {
        $merge = $this->taskMergeXmlReports();
        for ($i = 1; $i <= 5; $i++) {
            $merge->from(__DIR__ . "/tests/_output/result_$i.xml");
        }
        $merge->into(__DIR__ . '/tests/_output/result_paracept.xml')->run();
    }

    public function parallelRunAndMergeResults(): Result
    {
        $result = $this->parallelRun();
        $this->parallelMergeResults();

        return $result;
    }
}

Details


```yml

actor: AcceptanceTester
modules:
    enabled:
        - WebDriver:
           url: 'http://ci-office.statscore.com/'
           window_size: false
           log_js_errors: true
           port: 4444
           browser: chrome
           capabilities:
               chromeOptions:
                  args: ["--window-size=1600,900", "--headless"]
        - \Helper\Acceptance
        - Db
        - DbHelper
        - Laravel5:
            part: ORM
    config:
        Db:
          dsn: 'mysql:host=localhost;dbname=head'
          user: ''
          password: ''
          populate: false
          cleanup: false
extensions:
    enabled:
        - Codeception\Extension\RunProcess:
            0: exec docker run --net=host selenium/standalone-chrome
            sleep: 7 # wait 7 second for processes to boot
greg-1-anderson commented 4 years ago

Robo no longer uses Codeception for its internal tests. PRs to fix the Codeception tasks would be most welcome.