mozilla / frost

Unit testing framework for test driven security of AWS, GCP, Heroku and more.
https://mozilla.github.io/frost/
Mozilla Public License 2.0
104 stars 24 forks source link

Make sure all test id's are reproducible #412

Closed ajvb closed 3 years ago

ajvb commented 3 years ago

Currently, there are a number of tests where part of the test id / resource id is random. This is usually due to the @pytest.mark.parametrize call having two resources but the id function only support one of them, i.e. https://github.com/mozilla/frost/blob/master/aws/iam/test_iam_user_without_mfa.py#L8-L12

This leads to parametrize id's looking something like this: test_iam_user_without_mfa[ajvb-iam_user_mfa_devices9]

In this case, the number at the end of iam_user_mfa_devices9 is based on the order in which the test ran, meaning it is likely to change in the next run.

At the moment, this is not breaking anything, but it is a blocker for having state management around alerts / bugs created from results. Converting that test name and resource id into something key-able would require finicky custom logic.

Instead, all parametrize id's should be reproducible, so having it as test_iam_user_without_mfa[ajvb] or test_iam_user_without_mfa[ajvb-<constant-device-id>]

ajvb commented 3 years ago

List of tests that have this issue (for tracking in branch):