aws-ia / taskcat

Test all the CloudFormation things! (with TaskCat)
https://aws-ia.github.io/taskcat/
Apache License 2.0
1.17k stars 213 forks source link

`taskcat lint` uses all regions defined in `.taskcat.yaml` but doesn't output the region name in case of an error or warning #814

Open wombelix opened 1 year ago

wombelix commented 1 year ago

Describe the bug I have multiple regions defined in .taskcat.yaml, let's say us-east-1 on project level and us-east-2, us-west-1 and us-west-2 specifics tests. taskcat lint seem to use all regions defined in any level but if an error or warning occurs, the region name is not part of the output. In my case, I have a list of allowed EC2 Instances, all of them are supported in us-east-1, us-east-2 and us-west-2, but some are not available in us-west-1. In that case, taskcat lint throws a lot of warnings but I can't see in the output, that it's about the region us-west-1, which makes troubleshooting a little complicated.

To Reproduce Steps to reproduce the behavior:

  1. Are you testing a QuickStart or Custom template? - Partner Solution (in development)
  2. Attach or link a copy of the template if possible (remove any sensitive info) - Not public yet
  3. Provide the parameters that you passed. (remove any sensitive info) - see end of the issue
  4. How did you install taskcat? (docker or pip3) - pip
  5. Are you using a profile, an instance role or access keys to run taskcat? - profile
  6. Is your AWS environment configured via aws configure? - no, isengard

Expected behavior I would expect that taskcat provides the name of the region a specific error or warning belongs to. Even better would be, if the region can be set by a cli parameter or only regions on project level / the default test is taken into account if not configured otherwise.

Screenshots n/a

**Version (Please make sure you are running the latest version of taskcat)

Additional context I can technically configure to ignore instance type related warnings, but in case of a typo or something, it feels valuable to run an actual test. But 100% coverage of all instance types across all regions is not realistic, so testing just against one specific region, as sanity check that in general the list is correct, feels like a good approach.

Passed parameters

This is an example of a instance type allowed values list that works for example with us-east-1, because c6id is available in that region, and will throw warnings with us-west-1, which is expected and fine. But that I don't seem to have visibility in the taskcat lint output and can't limit the regions that should be included is something that could be improved.

Parameters:
  InstanceType:
    AllowedValues:
      - c6id.large
      - c6id.xlarge
      - c6id.2xlarge
      - c6id.4xlarge
      - c6id.8xlarge
      - c6id.12xlarge
      - c6id.16xlarge
      - c6id.24xlarge
      - c6id.32xlarge
      - c6id.metal

Warning

[WARN   ][WARN   ] :     line 62 [2030] [Check if parameters have a valid value] You must specify a valid allowed value for InstanceType
                                                              (c6id.large). Valid values are ["c1.medium", "c1.xlarge",
                                                              "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c3.large", "c3.xlarge",
                                                              "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "c4.large", "c4.xlarge",
                                                              "c5.12xlarge", "c5.18xlarge", "c5.24xlarge", "c5.2xlarge",
                                                              "c5.4xlarge", "c5.9xlarge", "c5.large", "c5.metal", "c5.xlarge", ........... ]