Add ErrorCheck to all tests + semgrep

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

To facilitate running acceptance tests across partitions, tests should include an ErrorCheck. ErrorCheck will skip tests if they fail for unsupported features rather than code problems. See the Contributing Guide for more information.

Definition of Done

• [x] All service acceptance tests with a PreCheck also have an ErrorCheck:
  • [x] ec2 (package non-test imports 150)
    • [x] #18144
    • [x] #18145
    • [x] #18146
    • [x] #18147
    • [x] #18148
    • [x] #18149
    • [x] #18153
    • [x] #18154
    • [x] #18155
    • [x] #18156
    • [x] #18158
    • [x] #18160
    • [x] #18161
    • [x] #18162
    • [x] #18163
  • [x] waf (imports 32)
    • [x] #18178
    • [x] #18180
  • [x] apigateway (imports 29)
    • [x] #18181
    • [x] #18184
    • [x] #18185
  • [x] iam (imports 29)
    • [x] #18186
    • [x] #18188
    • [x] #18189
  • [x] rds (imports 26)
    • [x] #18166
    • [x] #18167
    • [x] #18168
  • [x] directconnect (imports 18)
    • [x] #18190
    • [x] #18191
  • [x] glue (imports 18)
    • [x] #18193
    • [x] #18194
  • [x] opsworks (imports 16)
    • [x] #18195
    • [x] #18196
  • [x] accessanalyzer (imports 1) (#18263)
  • [x] acm (imports 3) (#18258)
  • [x] acmpca (imports 5) (#18253)
  • [x] apigatewayv2 (imports 13) (#18201)
  • [x] applicationautoscaling (imports 3) (#18299)
  • [x] appmesh (imports 7) (#18230)
  • [x] appsync (imports 5) (#18249)
  • [x] athena (imports 3) (#18260)
  • [x] autoscaling (imports 11) (#18165)
  • [x] autoscalingplans (imports 1) (#18299)
  • [x] backup (imports 10) (#18217)
  • [x] batch (imports 5) (#18254)
  • [x] budgets (imports 1) (#18299)
  • [x] cloud9 (imports 1) (#18266)
  • [x] cloudformation (imports 6) (#18244)
  • [x] cloudfront (imports 9)
  • [x] cloudhsmv2 (imports 3) (#18268)
  • [x] cloudtrail (imports 1) (#18262)
  • [x] cloudwatch (imports 3) (#18261)
  • [x] cloudwatchevents (imports 5) (#18296)
  • [x] cloudwatchlogs (imports 7) (#18295)
  • [x] codeartifact (imports 6) (#18245)
  • [x] codebuild (imports 4) (#18270)
  • [x] codecommit (imports 3) (#18264)
  • [x] codedeploy (imports 3) (#18267)
  • [x] codepipeline (imports 2) (#18265)
  • [x] codestarconnections (imports 1) (#18299)
  • [x] codestarnotifications (imports 1) (#18299)
  • [x] cognitoidentity (imports 2) (#18299)
  • [x] cognitoidentityprovider (imports 8) (#18247)
  • [x] configservice (imports 10) (#18218)
  • [x] costandusagereportservice (imports 1) (#18299)
  • [x] databasemigrationservice (imports 6) (#18293)
  • [x] datapipeline (imports 1) (#18299)
  • [x] datasync (imports 7) (#18231)
  • [x] dax (imports 3) (#18269)
  • [x] devicefarm (imports 1) (#18271)
  • [x] directoryservice (imports 4) (#18294)
  • [x] dlm (imports 1) (#18299)
  • [x] docdb (imports 7) (#18232)
  • [x] dynamodb (imports 5) (#18255)
  • [x] ecr (imports 6) (#18246)
  • [x] ecrpublic (imports 1) (#18272)
  • [x] ecs (imports 9) (#18226)
  • [x] efs (imports 8) (#18229)
  • [x] eks (imports 4) (#18297)
  • [x] elasticache (imports 7) (#18233)
  • [x] elasticbeanstalk (imports 6) (#18292)
  • [x] elasticsearchservice (imports 3) (#18299)
  • [x] elastictranscoder (imports 2) (#18299)
  • [x] elb (imports 12) (#18212)
  • [x] elbv2 (imports 10) (#18219)
  • [x] emr (imports 5) (#18256)
  • [x] firehose (imports 1) (#18299)
  • [x] fms (imports 2) (#18299)
  • [x] fsx (imports 2) (#18299)
  • [x] gamelift (imports 4) (#18284)
  • [x] glacier (imports 2) (#18300)
  • [x] globalaccelerator (imports 3) (#18300)
  • [x] guardduty (imports 10) (#18221)
  • [x] identitystore (imports 2) (#18300)
  • [x] imagebuilder (imports 12) (#18213)
  • [x] inspector (imports 4) (#18287)
  • [x] iot (imports 9) (#18228)
  • [x] kafka (imports 5) (#18257)
  • [x] kinesis (imports 2) (#18301)
  • [x] kinesisanalytics (imports 1) (#18301)
  • [x] kinesisanalyticsv2 (imports 1) (#18301)
  • [x] kinesisvideo (imports 1) (#18301)
  • [x] kms (imports 10) (#18222)
  • [x] lakeformation (imports 6) (#18252)
  • [x] lambda (imports 13) (#18208)
  • [x] lexmodelbuildingservice (imports 8) (#18285)
  • [x] licensemanager (imports 2) (#18302)
  • [x] lightsail (imports 5) (#18259)
  • [x] macie (imports 2) (#18302)
  • [x] mediaconvert (imports 1) (#18303)
  • [x] mediapackage (imports 1) (#18303)
  • [x] mediastore (imports 2) (#18303)
  • [x] mq (imports 3) (#18164)
  • [x] neptune (imports 9) (#18228)
  • [x] networkfirewall (imports 5) (#18286)
  • [x] organizations (imports 7) (#18241)
  • [x] outposts (imports 6) (#18251)
  • [x] pinpoint (imports 11) (#18216)
  • [x] pricing (imports 1) (#18302)
  • [x] prometheusservice (imports 1) (#18302)
  • [x] qldb (imports 2) (#18250)
  • [x] quicksight (imports 2) (#18302)
  • [x] ram (imports 5) (#18288)
  • [x] redshift (imports 10) (#18223)
  • [x] resourcegroups (imports 1) (#18302)
  • [x] route53 (imports 11) (#18057, #18248)
  • [x] route53resolver (imports 9) (#18248)
  • [x] s3 (imports 14) (#18197)
  • [x] s3/s3manager (imports 2)
  • [x] s3control (imports 5) (#18290)
  • [x] s3outposts (imports 1) (#18304)
  • [x] sagemaker (imports 14) (#18198)
  • [x] secretsmanager (imports 7) (#18240)
  • [x] securityhub (imports 7) (#18242)
  • [x] serverlessapplicationrepository (imports 1) (#18304)
  • [x] servicecatalog (imports 1) (#18302)
  • [x] servicediscovery (imports 4) (#18298)
  • [x] servicequotas (imports 3) (#18302)
  • [x] ses (imports 14) (#18200)
  • [x] sfn (imports 4) (#18237)
  • [x] shield (imports 1) (#18304)
  • [x] signer (imports 5) (#18273)
  • [x] simpledb (imports 1) (#18304)
  • [x] sns (imports 6) (#18234)
  • [x] sqs (imports 4) (#18236)
  • [x] ssm (imports 13) (#18209)
  • [x] ssoadmin (imports 6) (#18235)
  • [x] storagegateway (imports 10) (#18224)
  • [x] sts (imports 1) (#18304)
  • [x] swf (imports 1) (#18304)
  • [x] synthetics (imports 1) (#18304)
  • [x] transfer (imports 4) (#18291)
  • [x] wafregional (imports 13) (#18210)
  • [x] wafv2 (imports 10) (#18225)
  • [x] worklink (imports 2) (#18304)
  • [x] workspaces (imports 7) (#18243)
  • [x] xray (imports 3) (#18304)
• [x] Phase 2 (Missed in first pass, or part of other PRs) (See #18306, #18307, #18308, #18309)
  • [x] core_acceptance_test.go (1)
  • [x] data_source_aws_apigatewayv2_api_test.go (2)
  • [x] data_source_aws_arn_test.go (1)
  • [x] data_source_aws_billing_service_account_test.go (1)
  • [x] data_source_aws_cloudfront_cache_policy_test.go (1)
  • [x] data_source_aws_cloudfront_distribution_test.go (1)
  • [x] data_source_aws_cloudfront_origin_request_policy_test.go (1)
  • [x] data_source_aws_cloudtrail_service_account_test.go (2)
  • [x] data_source_aws_cloudwatch_log_group_test.go (4)
  • [x] data_source_aws_cur_report_definition_test.go (2)
  • [x] data_source_aws_ec2_transit_gateway_route_tables_test.go (2)
  • [x] data_source_aws_eks_cluster_auth_test.go (1)
  • [x] data_source_aws_elastic_beanstalk_hosted_zone_test.go (2)
  • [x] data_source_aws_elasticache_cluster_test.go (1)
  • [x] data_source_aws_elb_hosted_zone_id_test.go (1)
  • [x] data_source_aws_elb_service_account_test.go (2)
  • [x] data_source_aws_iam_policy_document_test.go (14)
  • [x] data_source_aws_iam_policy_test.go (1)
  • [x] data_source_aws_ip_ranges_test.go (2)
  • [x] data_source_aws_kms_secret_test.go (1)
  • [x] data_source_aws_partition_test.go (1)
  • [x] data_source_aws_redshift_service_account_test.go (2)
  • [x] data_source_aws_sagemaker_prebuilt_ecr_image_test.go (2)
  • [x] provider_test.go (19)
  • [x] resource_aws_apigatewayv2_api_mapping_test.go (4)
  • [x] resource_aws_cloudfront_cache_policy_test.go (3)
  • [x] resource_aws_cloudfront_distribution_test.go (28)
  • [x] resource_aws_cloudfront_origin_access_identity_test.go (3)
  • [x] resource_aws_cloudfront_origin_request_policy_test.go (3)
  • [x] resource_aws_cloudfront_public_key_test.go (3)
  • [x] resource_aws_cloudfront_realtime_log_config_test.go (3)
  • [x] resource_aws_customer_gateway_test.go (5)
  • [x] resource_aws_db_parameter_group_test.go (1)
  • [x] resource_aws_default_route_table_test.go (3)
  • [x] resource_aws_default_security_group_test.go (2)
  • [x] resource_aws_default_vpc_test.go (1)
  • [x] resource_aws_dms_replication_instance_test.go (1)
  • [x] resource_aws_ec2_carrier_gateway_test.go (3)
  • [x] resource_aws_ec2_client_vpn_network_association_test.go (1)
  • [x] resource_aws_ec2_tag_test.go (3)
  • [x] resource_aws_egress_only_internet_gateway_test.go (1)
  • [x] resource_aws_eip_test.go (11)
  • [x] resource_aws_elb_attachment_test.go (2)
  • [x] resource_aws_elb_test.go (19)
  • [x] resource_aws_iam_group_policy_test.go (2)
  • [x] resource_aws_iam_instance_profile_test.go (1)
  • [x] resource_aws_iam_role_policy_test.go (2)
  • [x] resource_aws_iam_role_test.go (2)
  • [x] resource_aws_iam_user_policy_test.go (2)
  • [x] resource_aws_instance_test.go (23)
  • [x] resource_aws_internet_gateway_test.go (3)
  • [x] resource_aws_key_pair_test.go (1)
  • [x] resource_aws_nat_gateway_test.go (1)
  • [x] resource_aws_network_acl_test.go (11)
  • [x] resource_aws_network_interface_attachment_test.go (1)
  • [x] resource_aws_network_interface_test.go (9)
  • [x] resource_aws_opsworks_stack_test.go (1)
  • [x] resource_aws_route_table_test.go (9)
  • [x] resource_aws_security_group_test.go (10)
  • [x] resource_aws_subnet_test.go (10)
  • [x] resource_aws_vpc_endpoint_connection_notification_test.go (1)
  • [x] resource_aws_vpc_endpoint_test.go (1)
  • [x] resource_aws_vpc_peering_connection_test.go (8)
  • [x] resource_aws_vpc_test.go (7)
  • [x] resource_aws_vpn_connection_test.go (6)
  • [x] resource_aws_vpn_gateway_route_propagation_test.go (1)
  • [x] resource_aws_vpn_gateway_test.go (4)
• [x] CI enforces having an ErrorCheck with tfproviderlint/XAT001 (#18309)

References

• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#TestCase.ErrorCheck
• hashicorp/terraform-plugin-sdk#592

• 17566

• 18057

• 18059

[bflad]

tfproviderlint has a new check, XAT001, which can detect this automatically:

...
Error: /home/runner/work/terraform-provider-aws/terraform-provider-aws/aws/data_source_aws_acm_certificate_test.go:38:36: XAT001: missing ErrorCheck
37  
38      resource.ParallelTest(t, resource.TestCase{
39          PreCheck:  func() { testAccPreCheck(t) },
Error: /home/runner/work/terraform-provider-aws/terraform-provider-aws/aws/data_source_aws_acm_certificate_test.go:110:36: XAT001: missing ErrorCheck
109 
110     resource.ParallelTest(t, resource.TestCase{
111         PreCheck:  func() { testAccPreCheck(t) },
Error: /home/runner/work/terraform-provider-aws/terraform-provider-aws/aws/data_source_aws_acm_certificate_test.go:158:36: XAT001: missing ErrorCheck
157 
158     resource.ParallelTest(t, resource.TestCase{
159         PreCheck:  func() { testAccPreCheck(t) },
Error: /home/runner/work/terraform-provider-aws/terraform-provider-aws/aws/data_source_aws_acm_certificate_test.go:197:36: XAT001: missing ErrorCheck
196 
197     resource.ParallelTest(t, resource.TestCase{
198         PreCheck:  func() { testAccPreCheck(t) },
Error: /home/runner/work/terraform-provider-aws/terraform-provider-aws/aws/data_source_aws_acmpca_certificate_authority_test.go:14:36: XAT001: missing ErrorCheck
13  
14      resource.ParallelTest(t, resource.TestCase{
15          PreCheck:  func() { testAccPreCheck(t) },
...

It will remain disabled in our makefile until we are ready.

[YakDriver]

Cool! Thanks. We have a lot of data sources and resources!

[ghost]

This has been released in version 3.34.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!