Implement a basic user management

[evangelion1204]

Many pages need user data e.g.login or register.

It should be possible to specify users in the behat.yml and use this saved data to fill forms or write custom steps:

default:
    QATools\BehatExtension:
      users:
        registered:
          firstname: michael
          lastname: geppert

public function fillLoginForm()
{
    $user = $this->getUser('registered');
    $this->loginForm->fill($user->getData());
}

public function enterFirstname()
{
    $user = $this->getUser('registered');
    $this->firstname->setValue($user->get('firstname'));
}

Normally only one user is active at time. Therefore extend the whole feature with an active user and specific steps to load him.

For example:

Scenario: A basic registration of an user
    Given "valid" as user

[aik099]

Considering that in Behat (in contrast to PHPUnit) you only have scenario + featurecontext + config, then the only place to emulate data sets would be config.

Dirty, but I don't see other options yet. You can define csv file and parse it to get desired users I guess.

[evangelion1204]

I currently use a php config for users and addresses. It would also possible to just specify a path for an additional config file where those information are loaded.

[aik099]

That sounds more like a Behat feature, not the QA-Tools extension. But anyway we start it here and then push as PR to Behat itself.

[evangelion1204]

I think extensions capable of solving things like this. We have our own context and can provide more functions. It would maybe be a good idea to move this feature to QA-Tools itself. This is maybe something which is needed quite often. We could also connect this with Form in the future.

[aik099]

Ability to separate data, used in tests from test code is something that test framework does. For example PHPUnit can do this. I don't know why this isn't implemented in Behat.

If you need these users only for single scenario, then TableNode ( http://blog.whiteoctober.co.uk/2012/09/12/behat-tablenodes-the-missing-manual/ ) might solve that. Maybe these table nodes can be moved in config and included in scenarios somehow as well.

[aik099]

Also @stof has suggested a solution on Twitter: http://behat.readthedocs.org/en/latest/guides/1.gherkin.html#scenario-outlines

[aik099]

Actually these Scenario Outlines complete eliminate the need to this task.

[evangelion1204]

I already read those a while ago, sure this can be used but I don't think that they are suiteable.

As a programmer they might be ok but normally tests will look like this:

Scenario: A basic login
  Given a registered user
  When the user fills the login form
  And clicks login
  Then the user will be on the account page

or

Scenario: A basic login which fails
  Given a non registered user
  When the user fills the login form
  And clicks login
  Then the user will be on the login page

This is how I have seen tests with Gherkin and Cucumber but maybe they were just bad examples for some sort of tests, yet I know that business guys wants to read those tests and understand whats happening.

Tests should also work as stand alone. The project I write tests for I need to login every time. Which results in repeating myself again and again, you could use http://docs.behat.org/guides/1.gherkin.html#backgrounds to repeat less but after all I want also to split my tests in separate files like:

• login
• register
• catalog
• product detail page
• checkout Also I don't want to write such data in the feature files, cause those will change in every test run. I might want to increment the used email address example_user_20140707_1@domain.tld ...

Maybe there are solutions with Behat which I currently not know or not aware of.

In general those outlines are the same as data providers in PHPUnit and they work great for unit tests, but maybe not for such global data as users or addresses, which I will also change depending on my country. In that case I could just load a different config.

[aik099]

I think, that at least if we're talking about users, the role concept, introduced in Behat 3.0 might be used. You write scenario as usual (one extra step As an ...), but then run it against a registered user role. http://behat.readthedocs.org/en/latest/guides/5.suites.html?highlight=role

And with tests split into several files having central user registry is pain, even in PHPUnit.

[evangelion1204]

Ok, I will take a closer look at this new Role feature, maybe it covers everything.

[evangelion1204]

And with tests split into several files having central user registry is pain, even in PHPUnit.

For a normal unit test this is right, but I don't agree for system Selenium tests or maybe I did not stumble over any problem so far. Yes some centralised registry is not the best solution in many cases.

[aik099]

This is Twitter thread that might be of interest: https://twitter.com/aik099/status/486125097024258049

[evangelion1204]

Checked the Role feature in Behat. This is more a filter. It scans in the user story for As an ... and just executes all Scenarios that match the specific role. This could be used in connection with an exchangeable context to specify data in the context itself. Like RegisteredUserContext or NewUserContext. After all I think this could lead to a valid but as I think a very complicated solution.

But I really like this new filter features of Behat :smile:

But for my case I need this feature (I already implemented it in some form in my current tests). Most of our test engineers work with a similar syntax. Given xxx as user to have testing data injected into the tests.

I would also like to switch data for different runs and they should not be hardcoded in the tests or contexts (addresses might change for country to country, same for email addresses). One more problem is, that in contrast to PHPUnit I don't have a clean environment, old data is still present. To my pity its a real testing or live system we execute those tests on. This means I need to change the email address for every test, considering a registration test.

[aik099]

I would also like to switch data for different runs and they should not be hardcoded in the tests or contexts (addresses might change for country to country, same for email addresses).

Then you're cleared to make this dataset feature as part of extension, since it kind of falls out of the scope of Gherkin anyway.

This means I need to change the email address for every test, considering a registration test.

That's not hard. Just save registered user name/id to a property of feature context class and use it in upcoming steps.

---

There are 2 approaches (at least with PHPUnit) on how you approach test data in general:

1. have super clever pre-populated DB and hope that each test will be happy
2. make each test to create needed DB state for it and after it finishes do a cleanup (be careful when tests run in parallel)

I prefer 2nd one.

[evangelion1204]

I prefer 2nd one.

This would be the perfect world for me :smile:, but for our environment user data or orders are not saved in our databases, we use many web services which I can't control or cleanup, I can create, modify, but not delete, which is fine from the architecture, but not very nice for testing.

That's not hard. Just save registered user name/id to a property of feature context class and use it in upcoming steps.

Yes, saving them for the whole instance is a good solution, but sometimes Selenium tests tend to fail (timeout, network error or something unexpected, which was not the fault of the app) then you want to rerun the failed ones with the prepared data. But data was not persisted in the context and is lost.

[aik099]

... then you want to rerun the failed ones with the prepared data. But data was not persisted in the context and is lost.

That's an interesting topic on it's own to be able to re-run only failed tests after test suite ends.

[evangelion1204]

Thats why I run only specific tests with tags (register, login, then checkout and other parts) and check for fail and rerun.

[aik099]

Thats why I run only specific tests with tags (register, login, then checkout and other parts) and check for fail and rerun.

So if you have 10 tests with register tag and 1 of them fails, then you rerun 10 tests? If so, then it's not too resource-effective solution.

[evangelion1204]

yes, like this, but better than to rerun the whole suite (of maybe 1000 tests, just because 1 failed), but maybe Behat offers something like a rerun after fail which I don't know.

Using the new filter feature and suite definition can maybe help more.

[stof]

behat has a --rerun option, which will run only the scenario whihc failed in the previous run

[evangelion1204]

Thanks for the hint, I was never aware of this option.

[aik099]

behat has a --rerun option, which will run only the scenario whihc failed in the previous run

That's something new, thanks @stof. However if you run Behat from CI environment that marks build as failed based on exit code from vendor/bin/behat, then it won't be possible to unmark CI build as failed based on 2nd run of vendor/bin/behat --rerun.

[aik099]

I'm not sure about PHPUnit analog to --rerun option, but if that will be possible, then problem arise when you need to build coverage report. Without extra programming it won't be possible to merge 2 coverage reports (from original run and from rerun) and I think that merging won't produce valid result either.

[stof]

That's something new

It is not new. Behat has it since 2.1.0: https://github.com/Behat/Behat/commit/1fcc797d9b9d6083f177bc5036614f43031ea7d9

However if you run Behat from CI environment that marks build as failed based on exit code from vendor/bin/behat, then it won't be possible to unmark CI build as failed based on 2nd run of vendor/bin/behat --rerun

you could have a bash script running behat, then rerunning it if needed (i.e. if the run failed), and the exiting with the exit code of the --rerun run. This way, the CI server sees only a single exit code.

[aik099]

It is not new. Behat has it since 2.1.0: https://github.com/Behat/Behat/commit/1fcc797d9b9d6083f177bc5036614f43031ea7d9

:smile: , I meant new for me personally, not that it was added in recent Behat version.

you could have a bash script running behat, then rerunning it if needed (i.e. if the run failed), and the exiting with the exit code of the --rerun run. This way, the CI server sees only a single exit code.

Great.