search

pytest-dev / pytest-bdd

BDD library for the py.test runner
https://pytest-bdd.readthedocs.io/en/latest/
MIT License
1.3k stars 219 forks source link

Scenarios that start with an "And" do not really run and incorrectly result in a pass. #403

Open Richard944 opened 3 years ago

Richard944 commented 3 years ago

Basically the scenario below passes when it should fail.

 Scenario: Richard Experiment 12 bad/passes
      And we shall fail

I know its weird for a scenario to start with an "And" but sometimes people do that when their background has a "Given" step in it. At the very least a test like the one above should fail with a syntax exception. The issue I am reporting occurs regardless of the presence of a background in the feature file.

Here are some more details to help understand the problem.

These pass which is correct:

 Scenario: Richard Experiment 1 good/passes
      When we shall pass

 Scenario: Richard Experiment 5 good/passes
      Given we shall pass
      When we shall pass
      Then we shall pass

 Scenario: Richard Experiment 8 good/passes
      Given we shall pass
      When we shall pass
      Then we shall pass

These fail which is also correct:

 Scenario: Richard Experiment 2 good/fails
      When we shall pass
      Then we shall fail

 Scenario: Richard Experiment 4 good/fails
      When we shall pass
      And we shall fail

 Scenario: Richard Experiment 6 good/fails
      Given we shall pass
      When we shall pass
      Then we shall fail

 Scenario: Richard Experiment 9 good/fails
      Given we shall fail

 Scenario: Richard Experiment 10 good/fails
      When we shall fail

 Scenario: Richard Experiment 11 good/fails
      Then we shall fail

These pass but they should fail. This is why we have a bug:

 Scenario: Richard Experiment 3 bad/passes
      And we shall fail
      Then we shall pass

 Scenario: Richard Experiment 7 bad/passes
      And we shall pass
      When we shall pass
      Then we shall fail

Based on "Richard Experiment 7 bad/passes" I would say that having a scenario which starts with an "And" breaks the whole scenario. None of the steps get run.

Here is my step code:

 @given('we shall pass')
 @when('we shall pass')
 @then('we shall pass')
 def just_pass():
      """
      This test step always passes.
      """
      value = 1
      assert value

 @given('we shall fail')
 @when('we shall fail')
 @then('we shall fail')
 def just_fail():
      """
      This test step always fails.
      """
      value = 0
      assert value

I am running in a 32 bit python 3.7.5 virtual environment with:

pip==21.0.1 pytest-bdd==4.0.2 pylint==2.6.0 twine==3.3.0 pytest==6.2.2 pytest-html==3.1.1 pytest-cov==2.11.1 pipdeptree==2.0.0 pyfakefs==4.3.3

Thanks!

youtux commented 3 years ago

I think that the parser treats everything within the Scenario as a "description", until the first "Given" is encountered. We should make sure that there is at least a "When" or a "Then" (or both) in each scenario, otherwise we should raise an error.

Richard944 commented 3 years ago

I think that there is a little more to it. In this scenario every step should cause a test failure but the test passes. The steps are not running even though there are "Given", "When" and "Then" steps.

Scenario: Richard Experiment 13 bad/passes
      And we shall fail
      Given we shall fail
      When we shall fail
      Then we shall fail

This scenario fails as it should.

Scenario: Richard Experiment 14 good/fails
      Given we shall fail
      When we shall fail
      Then we shall fail

I believe that something about starting with an "And" makes pytest-bdd think that the test has no steps at all.

VitaliyaIoffe commented 2 years ago

I have the similar issue

If we have something like the next scenario

Scenario: Scenario name
      And the first step
      And the second step

It raises 3 tests:

All these steps are empty and there is no error.

pytest-bdd==5.0.0