terraform-compliance / cli

a lightweight, security focused, BDD test framework against terraform.
https://terraform-compliance.com
MIT License
1.34k stars 151 forks source link

Using Background with 'when' and Scenario Outline together throws an error #599

Open mungojam opened 2 years ago

mungojam commented 2 years ago

Description

If I have a Given-When condition in a Background step and I add a Scenario Outline, then an error is thrown:

To Reproduce

Use the same terraform and plan.json as in https://github.com/terraform-compliance/cli/issues/498#issuecomment-1038233721) and then try the following feature file:

Feature: aa

    Background: bb
        Given I have aws_iam_openid_connect_provider defined
        When its url is https://dummy

    Scenario Outline: tt
        Given I have aws_iam_policy_document data configured

The error output is:

terraform-compliance --planfile plan.out --features ./features terraform-compliance v1.3.31 initiated

. Converting terraform plan file. 🚩 Features : /.../features2/ 🚩 Plan File : /.../plan.out.json

🚩 Running tests. 🎉

Feature: aa # /.../test.feature

Background: bb
    Given I have aws_iam_openid_connect_provider defined
    When its url is https://dummy

Scenario Outline: tt
    Given I have aws_iam_openid_connect_provider defined

Examples:
    |  |

1 features (1 passed) 0 scenarios (0 passed) 0 steps (0 passed) Run 1644941324 finished within a moment ❗ ERROR: Hook 'parse_in_step_variables' from /.../terraform_compliance/steps/terrain.py:14 raised: 'AttributeError: 'Context' object has no attribute 'bad_tags''

Traceback (most recent call last): File "/.../python3.8/site-packages/radish/hookregistry.py", line 132, in call func(model, *args, **kwargs) File "/.../python3.8/site-packages/terraform_compliance/steps/terrain.py", line 24, in parse_in_step_variables if step.context.bad_tags: AttributeError: 'Context' object has no attribute 'bad_tags'

mungojam commented 2 years ago

I forgot to add, if I take out the When condition from the Background then it works fine. And it works fine for Scenario rather than Scenario Outline too

mungojam commented 2 years ago

I tried to workaround this in two different ways, but both failed in another way (same as each other):

  1. Move the background to the beginning of each of the Scenario Outline blocks. I thought this would work as the docs suggest that a Given statement resets the context. However, I found that it then skipped the check altogether based on Given I have aws_iam_policy_document data configured, which was the first step of my actual check (sorry confused matters as I use a different example above)
  2. Change from using Background to using pre-conditions. This gave the same issue as the first workaround, where it could no longer find my aws_iam_policy_document that was in my first Given.

I haven't yet found a working workaround.

eerkunt commented 2 years ago

Sorry for bumping into this problem Mark. I will have a look this problem within this week.

eerkunt commented 2 years ago

It works with @precondition but doesn't really with Background. It loses all the context when Background is used - which is not a precondition for a Given directive, but it is for When and Then as they need to know what to use, how to use.

Super weird. This could be an issue as the radish library we use is old and they made quite a big change before releasing 1.0.0. This will require deeper investigation, sorry will take time.

mungojam commented 2 years ago

It works with @precondition but doesn't really with Background. It loses all the context when Background is used - which is not a precondition for a Given directive, but it is for When and Then as they need to know what to use, how to use.

Super weird. This could be an issue as the radish library we use is old and they made quite a big change before releasing 1.0.0. This will require deeper investigation, sorry will take time.

Thanks Emre, I appreciate you looking into it when you get a chance. It's a bit of a side-project for me to get these new checks in.