This leads to parametrize id's looking something like this: test_iam_user_without_mfa[ajvb-iam_user_mfa_devices9]
In this case, the number at the end of iam_user_mfa_devices9 is based on the order in which the test ran, meaning it is likely to change in the next run.
At the moment, this is not breaking anything, but it is a blocker for having state management around alerts / bugs created from results. Converting that test name and resource id into something key-able would require finicky custom logic.
Instead, all parametrize id's should be reproducible, so having it as test_iam_user_without_mfa[ajvb] or test_iam_user_without_mfa[ajvb-<constant-device-id>]
Currently, there are a number of tests where part of the test id / resource id is random. This is usually due to the
@pytest.mark.parametrize
call having two resources but the id function only support one of them, i.e. https://github.com/mozilla/frost/blob/master/aws/iam/test_iam_user_without_mfa.py#L8-L12This leads to parametrize id's looking something like this:
test_iam_user_without_mfa[ajvb-iam_user_mfa_devices9]
In this case, the number at the end of
iam_user_mfa_devices9
is based on the order in which the test ran, meaning it is likely to change in the next run.At the moment, this is not breaking anything, but it is a blocker for having state management around alerts / bugs created from results. Converting that test name and resource id into something key-able would require finicky custom logic.
Instead, all parametrize id's should be reproducible, so having it as
test_iam_user_without_mfa[ajvb]
ortest_iam_user_without_mfa[ajvb-<constant-device-id>]