Behat testing setup in docs doesn't work without modification

[geerlingguy]

I've followed the steps in the Testing and Specifying the API guide, but it looks like there's some auto-wiring missing that is necessary to get everything working. At minimum, maybe the addition of a behat.yml file to specify contexts? I'm not seeing how Behat is supposed to see behatch/contexts unless that package is also installed.

Basically, I get:

4 scenarios (4 undefined)
37 steps (37 undefined)
0m0.14s (7.94Mb)

 >> default suite has undefined steps. Please choose the context to generate snippets:

  [0] None
  [1] FeatureContext

When I run docker-compose exec php vendor/bin/behat.

I also noticed https://github.com/api-platform/api-platform/issues/742, which seems to mention adding more configuration to get doctrine to see the SQLite database. What isn't clear to me, though, is how to use behatch/contexts without also adding in Mink and potentially other deps to be able to hit URL endpoints.

[geerlingguy]

So next step, I grabbed the example behat.yml from the demo project: https://github.com/api-platform/demo/blob/master/api/behat.yml.dist

I placed it in api/behat.yml, and then ran:

docker-compose exec php composer require --dev behatch/contexts behat/mink behat/symfony2-extension

And then ran docker-compose exec php vendor/bin/behat, and get:

In SymfonyFactory.php line 54:

  Install MinkBrowserKitDriver in order to use the symfony2 driver. 

So now, docker-compose exec php composer require --dev behat/mink-browserkit-driver to grab that, and then after running Behat again:

$ docker-compose exec php vendor/bin/behat
Feature:
  In order to prove that the Behat Symfony extension is correctly installed
  As a user
  I want to have a demo scenario

In ConstructorArgumentOrganiser.php line 81:

  `FeatureContext::__construct()` does not expect argument `$doctrine`. 

So I set that context to just:

  suites:
    default:
      contexts:
        - FeatureContext

And now it seems to run the tests—but it's running them against the pgsql database. So next step is I'll look again at #742 and see if I can get it to use the sqlite database.

[geerlingguy]

So, I have a working setup, following partly along with the steps in #742:

api/behat.yml contents:

default:
  calls:
    error_reporting: 16383 # E_ALL & ~E_USER_DREPRECATED
  suites:
    default:
      contexts:
        - FeatureContext
        - Behat\MinkExtension\Context\MinkContext
        - Behatch\Context\RestContext
        - Behatch\Context\JsonContext
  extensions:
    Behat\Symfony2Extension:
      kernel:
        bootstrap: "features/bootstrap/bootstrap.php"
        class: "App\\Kernel"
    Behat\MinkExtension:
      base_url: "http://localhost/"
      sessions:
        default:
          symfony2: ~
    Behatch\Extension: ~

api/composer.json scripts entry:

        "behat": [
            "bin/console c:c --env=test",
            "bin/console d:d:c --env=test",
            "bin/console d:s:d -f --env=test",
            "bin/console d:s:u -f --env=test",
            "APP_ENV=test behat --colors"
        ]

Then run:

docker-compose exec php composer run-script behat

I can get the tests to pass locally, but not when I run in the CI environment under Travis CI. It's so strange, because the deps are locked in using Composer, and the environment is all contained within Docker, so it should be identical :-/

On CI I get:

In ConstructorArgumentOrganiser.php line 81:

  `FeatureContext::__construct()` does not expect argument `$kernel`.  

[geerlingguy]

Ah. Didn't realize there was already a behat.yml.dist included in the repo (and that my behat.yml was not being committed at all!).

I committed my behat.yml, and now tests are passing in Travis CI.

So next step—should this be closed as 'works as designed', or is the intent of the testing docs page to go from 'new API project' to 'tests working with Behat'? Because right now there are a number of setup steps left out of that doc which seem necessary to be able to do any testing or follow the example.

[dunglas]

Thanks for sharing your investigations! This is a known issue that this page is broken (since the Symfony 4 release actually...). I started to fix it but I had not the time to finish yet: https://github.com/api-platform/docs/pull/535

Would you mind opening a doc PR (at first) to fix the tutorial with what you explained here? It would be a very good first step!

[dunglas]

Ah. Didn't realize there was already a behat.yml.dist included in the repo (and that my behat.yml was not being committed at all!).

It’s the default setup of the Flex recipe for Behat. Your default config should go in this file, and only the local overrides should be in behat.yml

[dunglas]

For the sake of completeness, it is broken because API Platform was shipped with Behat and Behatch when it was based on the Symfony Standard Edition, but they have been removed form the new Flex-based skeleton and now need to be installed separately.

[geerlingguy]

@dunglas - Perfectly understandable, thanks for the clarification!

I'll take a crack at upgrading the docs, since it's still fresh and I'm guessing I'll have to refer to them again myself, next time I set up an API ;)

[ghost]

I can't get this setup to work for the life of me, followed all the steps here and in #742 and I still get data showing up in the dev database after POST requests, APP_ENV=test seems to have no effect because if I instead edit packages/doctrine.yaml to use the sqllite database, behat starts throwing an error of "Operation 'Doctrine\DBAL\Platforms\AbstractPlatform::getSequenceNextValSQL' is not supported by platform". I don't know why it's so hard to test this thing

Update (fixed):

At some point I had configured MinkExtension to use Goutte instead of symfony2, changing it to symfony2 somehow fixed the issue.

[Cochonours]

I'm trying to create a symfony4+apiplatform project for the first time and this is driving me insane. I would like to be able to define a few tests for the api, and am completely stuck after trying to follow all the possible options given by my google search. I installed all the things @geerlingguy mentionned

Here are the relevant parts of my composer.json

    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "App\\Tests\\": "tests/"
        }
    },
    "require": {
        "php": ">=7.2.0",
        "api-platform/api-pack": "^1.1",
        "doctrine/doctrine-bundle": "^1.6",
        "doctrine/orm": "^2.5",
        "incenteev/composer-parameter-handler": "^2.0",
        "sensio/framework-extra-bundle": "^5.2",
        "symfony/apache-pack": "^1.0",
        "symfony/asset": "^4.1",
        "symfony/console": "^3.3",
        "symfony/dotenv": "^4.1",
        "symfony/flex": "^1.1",
        "symfony/form": "^4.1",
        "symfony/monolog-bundle": "^3.3",
        "symfony/orm-pack": "^1.0",
        "symfony/polyfill-apcu": "^1.0",
        "symfony/security-bundle": "^4.1",
        "symfony/swiftmailer-bundle": "^3.2",
        "symfony/templating": "^4.1",
        "symfony/translation": "^4.1",
        "symfony/twig-bundle": "^4.1",
        "symfony/validator": "^4.1"
    },
    "require-dev": {
        "behat/behat": "^3.5",
        "behat/gherkin": "^4.5",
        "behat/mink": "^1.7",
        "behat/mink-browserkit-driver": "^1.3",
        "behat/mink-extension": "^2.3",
        "behat/symfony2-extension": "^2.1",
        "behatch/contexts": "^3.2",
        "phpunit/phpunit": "^7.4",
        "sensiolabs/security-checker": "^5.0",
        "symfony/debug-bundle": "^4.1",
        "symfony/phpunit-bridge": "^3.0",
        "symfony/web-profiler-bundle": "^4.1"
    },

My config/behat.yml is basically the same as @geerlingguy but I need to add some obscure paths so the "features" folder would appear in "/tests" instead of the root folder.

default:
  calls:
    error_reporting: 16383 # E_ALL & ~E_USER_DREPRECATED
  suites:
    default:
      paths:
        features: 'tests/features'
        bootstrap: 'tests/features/bootstrap'
      contexts:
        - FeatureContext
        - Behat\MinkExtension\Context\MinkContext
        - Behatch\Context\RestContext
        - Behatch\Context\JsonContext
  gherkin:
    cache: ~
    filters:
      tags: ~@wip
  extensions:
    Behat\Symfony2Extension:
      kernel:
        bootstrap: "tests/features/bootstrap/bootstrap.php"
        class: "App\\Kernel"
    Behat\MinkExtension:
      base_url: "http://localhost:8081/jaguar2/api"
      sessions:
        default:
          symfony2: ~
Behatch\Extension: ~

My FeatureContext:

<?php

use Behat\Behat\Context\Context;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\KernelInterface;

/**
 * This context class contains the definitions of the steps used by the demo 
 * feature file. Learn how to get started with Behat and BDD on Behat's website.
 * 
 * @see http://behat.org/en/latest/quick_start.html
 */
class FeatureContext implements Context
{
    /**
     * @var KernelInterface
     */
    private $kernel;

    /**
     * @var Response|null
     */
    private $response;

    public function __construct(KernelInterface $kernel)
    {
        $this->kernel = $kernel;
    }
}

And when I run "vendor/bin/behat" I get Can not find a matching value for an argument `$request` of the method `Behatch\Context\RestContext::__construct()`. and have no idea what to do.

[Cochonours]

I think I may have found the problem. I do not know why (probably a "behat --init" while I was playing with the paths) but I saw I had a misplaced file lost in the config config/features/bootstrap/FeatureContext.php. I removed that and now I get the `FeatureContext` context class not found and can not be used.

So... how am I supposed to tell behat to search for that file in "/tests/features/bootstrap/FeatureContext.php", considering the config is in "/config/behat.yml"? I just tried updating the different paths I have in there with "../tests" instead of "tests" but to no avail.

[saamorim]

After a few steps I think I've manage to setup behat+behatch but some errors appear. To recap:

$ docker-compose exec php composer require --dev behatch/contexts behat/mink behat/symfony2-extension behat/mink-browserkit-driver
$ docker-compose exec php vendor/bin/behat -V
$ docker-compose exec php vendor/bin/behat --init

Added the api/behat.yaml with the content:

default:
  calls:
    error_reporting: 16383 # E_ALL & ~E_USER_DREPRECATED
  suites:
    default:
      contexts:
        - FeatureContext: { doctrine: "@doctrine" }
        - Behat\MinkExtension\Context\MinkContext
        - Behatch\Context\RestContext
        - Behatch\Context\JsonContext
  extensions:
    Behat\Symfony2Extension:
      kernel:
        env: 'test'
        debug: 'true'
        bootstrap: "features/bootstrap/bootstrap.php"
        class: "App\\Kernel"
    Behat\MinkExtension:
      base_url: "http://localhost/"
      sessions:
        default:
          symfony2: ~
    Behatch\Extension: ~

Added the api/features/bootstrap/bootstrap.php with the content

<?php

use Symfony\Component\Dotenv\Dotenv;

// The check is to ensure we don't use .env in production
if (!isset($_SERVER['APP_ENV'])) {
    if (!class_exists(Dotenv::class)) {
        throw new \RuntimeException('APP_ENV environment variable is not defined. You need to define environment variables for configuration or add "symfony/dotenv" as a Composer dependency to load variables from a .env file.');
    }
    (new Dotenv())->load(__DIR__.'/../../.env');
}

Change the api/features/bootstrap/FeatureContext.php to be:

<?php

use Behat\Behat\Context\Context;
use Behat\Behat\Context\SnippetAcceptingContext;
use Doctrine\Common\Persistence\ManagerRegistry;
use Doctrine\ORM\Tools\SchemaTool;
use Behat\Gherkin\Node\PyStringNode;
use Behat\Gherkin\Node\TableNode;

/**
 * Defines application features from the specific context.
 */
class FeatureContext implements Context, SnippetAcceptingContext
{
    /**
     * @var ManagerRegistry
     */
    private $doctrine;
    /**
     * @var SchemaTool
     */
    private $schemaTool;
    /**
     * @var array
     */
    private $classes;

    /**
     * Initializes context.
     *
     * Every scenario gets its own context instance.
     * You can also pass arbitrary arguments to the
     * context constructor through behat.yml.
     */
    public function __construct(ManagerRegistry $doctrine)
    {
        $this->doctrine = $doctrine;
        $manager = $doctrine->getManager();
        $this->schemaTool = new SchemaTool($manager);
        $this->classes = $manager->getMetadataFactory()->getAllMetadata();
    }

    /**
     * @BeforeScenario @createSchema
     */
    public function createDatabase()
    {
        $this->schemaTool->dropSchema($this->classes);
        $this->doctrine->getManager()->clear();
        $this->schemaTool->createSchema($this->classes);
    }
}

Finally added the content to the api/features/book.feature as stated in https://api-platform.com/docs/distribution/testing

After running behat

$ docker-compose exec php vendor/bin/behat 

got the output:

4 scenarios (1 passed, 3 failed)
27 steps (24 passed, 3 failed)
0m0.57s (30.32Mb)

All of the failures are due to the ordering of the id attribute. Changing the id's to the end fixed the issues

Other solution for fixing this issue is to define the order of the parameters using the order attribute (https://api-platform.com/docs/core/default-order)

<?php
// api/src/Entity/Book.php

namespace App\Entity;

use ApiPlatform\Core\Annotation\ApiResource;

/**
 * @ApiResource(attributes={"order"={"foo", "bar"}})
 */
class Book

[dannylewis-sheffield]

@saamorim guide just about worked for me but needed a few modifications.

The version of behat/mink needed to be "~1.7@dev"

https://api-platform.com/docs/core/default-order also doesn't work, this was their second proposal but I think this is for ordering the data not the structure, you need to change the order in the feature file and place the id attributes last.

Finally the createSchema and dropSchema annotation don't work, and it's using the main database and not a temporary SQLite database, as @geerlingguy says #742 might have some more answers on this.

Could this be given a higher priority? If I get the time I'll investigate further and submit a patch but right now I can run tests for the javascript side of projects, we are using your wonderful docker setup, but not the PHP and main API side of the project.

[dannylewis-sheffield]

As a side note with regard to the ordering of the json structure and failing tests, the id placement above, I think that the upstream behatch/contexts could be improved or this project could write an additional context, I'll raise an issue with them as well.

https://github.com/Behatch/contexts/blob/master/src/Context/JsonContext.php See method: public function theJsonShouldBeEqualTo

Could be amended to check if the objects are functionally the same

[dannylewis-sheffield]

I've changed the feature file comparisons from:

And the JSON should be equal to: to And the JSON should be functionally equal to:

and updated api/features/bootstrap/FeatureContext.php to be:

<?php

use Doctrine\Common\Persistence\ManagerRegistry;
use Doctrine\ORM\Tools\SchemaTool;
use Behat\Gherkin\Node\PyStringNode;
use Behatch\Json\JsonInspector;
use Behatch\HttpCall\HttpCallResultPool;
use Behatch\Json\Json;

/**
 * Defines application features from the specific context.
 */
class FeatureContext extends Behatch\Context\BaseContext
{
    /**
     * @var ManagerRegistry
     */
    private $doctrine;
    /**
     * @var SchemaTool
     */
    private $schemaTool;
    /**
     * @var array
     */
    private $classes;

    protected $inspector;
    protected $httpCallResultPool;

    /**
     * Initializes context.
     *
     * Every scenario gets its own context instance.
     * You can also pass arbitrary arguments to the
     * context constructor through behat.yml.
     */
    public function __construct(ManagerRegistry $doctrine, HttpCallResultPool $httpCallResultPool, $evaluationMode = 'javascript')
    {
        $this->doctrine = $doctrine;
        $manager = $doctrine->getManager();
        $this->schemaTool = new SchemaTool($manager);
        $this->classes = $manager->getMetadataFactory()->getAllMetadata();

        $this->inspector = new JsonInspector($evaluationMode);
        $this->httpCallResultPool = $httpCallResultPool;
    }

    /**
     * @BeforeScenario @createSchema
     */
    public function createDatabase()
    {
        $this->schemaTool->createSchema($this->classes);
    }

    /**
     * @AfterScenario @dropSchema
     */
    public function dropDatabase()
    {
        $this->schemaTool->dropSchema($this->classes);
    }

    /**
     * @Then the JSON should be functionally equal to:
     */
    public function theJsonShouldBeFunctionallyEqualTo(PyStringNode $content)
    {

        $actual = new Json($this->httpCallResultPool->getResult()->getValue());

        try {
            $expected = new Json($content);
        } catch (\Exception $e) {
            throw new \Exception('The expected JSON is not a valid');
        }

        $actualArray = (array) $actual->getContent();
        $expectedArray = (array) $expected->getContent();

        $this->ksortTree($actualArray);
        $this->ksortTree($expectedArray);

        $this->assertSame(
            (string) json_encode($expectedArray, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE),
            (string) json_encode($actualArray, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE),
            "The json is equal to:\n". $actual->encode()
        );

    }

    /**
     * @param array $array
     */
    private function ksortTree(&$array)
    {

        if(is_object($array)){
            $array = (array) $array;
        }

        if (!is_array($array)) {
            return false;
        }

        ksort($array);
        foreach ($array as $k=>$v) {
            $this->ksortTree($array[$k]);
        }
        return true;
    }

}

Still room for improvement but it's working

[saamorim]

Thank for the input @dannylewis-sheffield, recreate a new project followed the api-platform setup instructions, without creating the books feature, then followed my instructions with a few tweaks:

$ docker-compose exec php composer require --dev behatch/contexts behat/mink:"~1.7@dev" behat/symfony2-extension behat/mink-browserkit-driver
$ docker-compose exec php vendor/bin/behat -V
$ docker-compose exec php vendor/bin/behat --init

The api/behat.yaml now has the following content:

default:
  calls:
    error_reporting: 16383 # E_ALL & ~E_USER_DREPRECATED
  suites:
    default:
      contexts:
        - FeatureContext: { kernel: "@kernel", doctrine: "@doctrine" }
        - Behat\MinkExtension\Context\MinkContext
        - Behatch\Context\RestContext
        - Behatch\Context\JsonContext
  extensions:
    Behat\Symfony2Extension:
      kernel:
        env: 'test'
        debug: 'true'
        bootstrap: "features/bootstrap/bootstrap.php"
        class: "App\\Kernel"
    Behat\MinkExtension:
      base_url: "http://localhost/"
      sessions:
        default:
          symfony2: ~
    Behatch\Extension: ~

Feature context has the following:

<?php

use Behat\Behat\Context\Context;
use Doctrine\Common\Persistence\ManagerRegistry;
use Doctrine\ORM\Tools\SchemaTool;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\KernelInterface;
use Behat\Behat\Context\SnippetAcceptingContext;

/**
 * This context class contains the definitions of the steps used by the demo
 * feature file. Learn how to get started with Behat and BDD on Behat's website.
 *
 * @see http://behat.org/en/latest/quick_start.html
 */
class FeatureContext implements Context, SnippetAcceptingContext
{
    /**
     * @var KernelInterface
     */
    private $kernel;

    /**
     * @var Response|null
     */
    private $response;

    /**
     * @var ManagerRegistry
     */
    private $doctrine;
    /**
     * @var SchemaTool
     */
    private $schemaTool;
    /**
     * @var array
     */
    private $classes;

    public function __construct(KernelInterface $kernel, ManagerRegistry $doctrine)
    {
        $this->kernel = $kernel;
        $this->doctrine = $doctrine;
        $manager = $doctrine->getManager();
        $this->schemaTool = new SchemaTool($manager);
        $this->classes = $manager->getMetadataFactory()->getAllMetadata();
    }

    /**
     * @BeforeScenario @createSchema
     */
    public function createDatabase()
    {
        $this->schemaTool->dropSchema($this->classes);
        $this->doctrine->getManager()->clear();
        $this->schemaTool->createSchema($this->classes);
    }

    /**
     * @When a demo scenario sends a request to :path
     */
    public function aDemoScenarioSendsARequestTo(string $path)
    {
        $this->response = $this->kernel->handle(Request::create($path, 'GET'));
    }

    /**
     * @Then the response should be received
     */
    public function theResponseShouldBeReceived()
    {
        if ($this->response === null) {
            throw new \RuntimeException('No response received');
        }
    }
}

after running, got the following output

$ docker-compose exec php vendor/bin/behat
Feature:
  In order to prove that the Behat Symfony extension is correctly installed
  As a user
  I want to have a demo scenario

  Scenario: It receives a response from Symfony's kernel # features/demo.feature:10
    When a demo scenario sends a request to "/"          # FeatureContext::aDemoScenarioSendsARequestTo()
    Then the response should be received                 # FeatureContext::theResponseShouldBeReceived()

1 scenario (1 passed)
2 steps (2 passed)
0m0.23s (26.89Mb)

[shehi]

@saamorim , technically passing an ObjectManager instance to SchemaTool can be dangerous. SchemaTool only works with EntityManager (ORM tooling, nothing else).

[dunglas]

This docs have been removed in favor of the new testing tool, see https://github.com/api-platform/docs/pull/875