search

StyraInc / opa-aws-cloudformation-hook

AWS Cloudformation Hook for OPA-powered infrastructure policy enforcement
Apache License 2.0
35 stars 5 forks source link

Add integration tests #15

Closed anderseknert closed 2 years ago

anderseknert commented 2 years ago

WIP

Attempt to send each template file to OPA for testing.

To test, first start OPA:

opa run --server --log-format json-pretty --watch policy/

Then run the script:

./test-templates.py

Current result:

./test-templates.py
ERROR: Exception raised when loading templates/s3.yaml could not determine a constructor for the tag '!Ref'
  in "templates/s3.yaml", line 15, column 15
SUCCESS: templates/iam-users-have-policy/examples/iam-fail-no-user-policy.yaml
FAIL: templates/iam-users-have-policy/examples/iam-success-user-policy-attached.yaml
FAIL: templates/iam-users-have-policy/examples/iam-success-user-policy-attached.yaml
FAIL: templates/iam-no-admin-statements/examples/iam-success-action-restricted.yaml
SUCCESS: templates/iam-no-admin-statements/examples/iam-fail-admin-allowed.yaml
FAIL: templates/s3-bucket-logging-enabled/examples/s3-success-logging-enabled.yaml
SUCCESS: templates/s3-bucket-logging-enabled/examples/s3-fail-no-logging.yaml
ERROR: Exception raised when loading templates/s3-block-public-access/examples/s3-success-name-prefix-excluded.yaml could not determine a constructor for the tag '!Sub'
  in "templates/s3-block-public-access/examples/s3-success-name-prefix-excluded.yaml", line 5, column 19
ERROR: Exception raised when loading templates/s3-block-public-access/examples/s3-fail-no-settings-specified.yaml could not determine a constructor for the tag '!Sub'
  in "templates/s3-block-public-access/examples/s3-fail-no-settings-specified.yaml", line 5, column 19
ERROR: Exception raised when loading templates/s3-block-public-access/examples/s3-success-all-access-blocked.yaml could not determine a constructor for the tag '!Sub'
  in "templates/s3-block-public-access/examples/s3-success-all-access-blocked.yaml", line 5, column 19
FAIL: templates/s3-block-public-access/examples/s3-success-no-name-and-all-acces-blocked.yaml
ERROR: Exception raised when loading templates/s3-block-public-access/examples/s3-fail-not-all-4-settings-specified.yaml could not determine a constructor for the tag '!Sub'
  in "templates/s3-block-public-access/examples/s3-fail-not-all-4-settings-specified.yaml", line 5, column 19
SUCCESS: templates/s3-bucket-encryption/examples/s3-fail-wrong-encryption-type.yaml
SUCCESS: templates/s3-bucket-encryption/examples/s3-fail-wrong-encryption-type.yaml
SUCCESS: templates/s3-bucket-encryption/examples/s3-fail-no-bucket-encryption.yaml
ERROR: Exception raised when loading templates/s3-bucket-encryption/examples/s3-success-encryption.yaml could not determine a constructor for the tag '!Ref'
  in "templates/s3-bucket-encryption/examples/s3-success-encryption.yaml", line 36, column 12
ERROR: Exception raised when loading templates/rds-encryption-verify/examples/rds-success-encryption-set.yaml could not determine a constructor for the tag '!Or'
  in "templates/rds-encryption-verify/examples/rds-success-encryption-set.yaml", line 68, column 15
ERROR: Exception raised when loading templates/rds-encryption-verify/examples/rds-fail-encryption-not-set.yaml could not determine a constructor for the tag '!Or'
  in "templates/rds-encryption-verify/examples/rds-fail-encryption-not-set.yaml", line 68, column 15
SUCCESS: templates/security-group-open-ingress/examples/sg-success-restricted-to-subnet.yaml
SUCCESS: templates/security-group-open-ingress/examples/sg-fail-open-to-public.yaml
ERROR: Exception raised when loading templates/iam-principal-boundary/examples/iam-success-permission-boundary-included.yaml could not determine a constructor for the tag '!Sub'
  in "templates/iam-principal-boundary/examples/iam-success-permission-boundary-included.yaml", line 7, column 17
SUCCESS: templates/iam-principal-boundary/examples/iam-fail-permission-boundary-incorrect.yaml
ERROR: Exception raised when loading templates/iam-principal-boundary/examples/iam-fail-auto-generated-name-not-excluded.yaml could not determine a constructor for the tag '!Sub'
  in "templates/iam-principal-boundary/examples/iam-fail-auto-generated-name-not-excluded.yaml", line 6, column 17
ERROR: Exception raised when loading templates/iam-principal-boundary/examples/iam-success-role-name-excluded.yaml could not determine a constructor for the tag '!Sub'
  in "templates/iam-principal-boundary/examples/iam-success-role-name-excluded.yaml", line 6, column 17
ERROR: Exception raised when loading templates/iam-principal-boundary/examples/iam-success-user-name-excluded.yaml could not determine a constructor for the tag '!Sub'
  in "templates/iam-principal-boundary/examples/iam-success-user-name-excluded.yaml", line 6, column 17

As seen above, a lot of them fail due to the YAML parser not being able to parse the !Sub or !Ref expressions. Will need to pick up from there, and then address the tests that do parse but print FAIL.

Signed-off-by: Anders Eknert anders@eknert.com

peteroneilljr commented 2 years ago

I implemented a fix to skip the parsing of AWS references. You can read about it in the blog below.

https://death.andgravity.com/any-yaml

But the script is now failing somewhere else, I think it's having trouble looping through the rest of the files.