awscc_securityhub_finding_aggregator: change in region_linking_mode leads to failure when regions are listed in prior state

quixoticmonk commented 4 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

tf -version
Terraform v1.9.0
on darwin_amd64
+ provider registry.terraform.io/gavinbunney/kubectl v1.14.0
+ provider registry.terraform.io/hashicorp/aws v5.58.0
+ provider registry.terraform.io/hashicorp/awscc v1.6.0

Affected Resource(s)

awscc_securityhub_finding_aggregator

Terraform Configuration Files

Original configuration

resource "awscc_securityhub_finding_aggregator" "example" {
  region_linking_mode = "ALL_REGIONS_EXCEPT_SPECIFIED"
  regions = [
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ap-southeast-4"
  ]
}

Updated configuration


resource "awscc_securityhub_finding_aggregator" "example" {
  region_linking_mode = "ALL_REGIONS"
}

Debug Output

2024-07-22T00:34:08.290-0400 [DEBUG] provider.terraform-provider-awscc_v1.6.0_x5: 
Detected value change between proposed new state and prior state: tf_provider_addr=registry.terraform.io/hashicorp/awscc 
tf_req_id=fb120fdd-7d7b-cc32-a3a6-dc7727c47aad tf_rpc=PlanResourceChange 
tf_attribute_path=region_linking_mode 
tf_resource_type=awscc_securityhub_finding_aggregator @caller=github.com/hashicorp/terraform-plugin-framework@v1.10.0/internal/fwserver/server_planresourcechange.go:208 @module=sdk.framework timestamp=2024-07-22T00:34:08.290-0400

tf_provider_addr=registry.terraform.io/hashicorp/awscc @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.54/logging/tf_logger.go:45 http.method=POST
  http.request.body=
  | {"ClientToken":"terraform-20240722043409855000000001","Identifier":"arn:aws:securityhub:us-east-1:############:finding-aggregator/a8c86c57-dd4c-9a47-4a3d-93cab1506add","PatchDocument":"[{\"op\":\"replace\",\"path\":\"/RegionLinkingMode\",\"value\":\"ALL_REGIONS\"}]","TypeName":"AWS::SecurityHub::FindingAggregator"}

Planned state

{
  "finding_aggregation_region": "us-east-1",
  "finding_aggregator_arn": "arn:aws:securityhub:us-east-1:############:finding-aggregator/a8c86c57-dd4c-9a47-4a3d-93cab1506add",
  "id": "arn:aws:securityhub:us-east-1:############:finding-aggregator/a8c86c57-dd4c-9a47-4a3d-93cab1506add",
  "region_linking_mode": "ALL_REGIONS",
  "regions": [
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ap-southeast-4"
  ]
}

Panic Output

Expected Behavior

The security aggregator should be modified or replace to ensure all regions are covered.

Actual Behavior

The terraform apply step fails with the below error message


│ Error: AWS SDK Go Service Operation Incomplete
│
│   with awscc_securityhub_finding_aggregator.example,
│   on main.tf line 22, in resource "awscc_securityhub_finding_aggregator" "example":
│   22: resource "awscc_securityhub_finding_aggregator" "example" {
│
│ Waiting for Cloud Control API service UpdateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: Regions cannot be passed as input if
│ RegionLinkingMode is set to 'ALL_REGIONS' or 'NO_REGIONS'. (Service: AWSSecurityHub; Status Code: 400; Error Code: InvalidInputException; Request ID: 52f48476-eed4-4590-a09f-85f73a89b312;
│ Proxy: null). ErrorCode: InvalidRequest

Steps to Reproduce

terraform apply
terraform apply

Important Factoids

References

quixoticmonk commented 4 months ago

Digging into this deeper with the cloudcontrol API

Create resource with CC API


aws cloudcontrol create-resource \
  --region us-east-1 \
  --type-name "AWS::SecurityHub::FindingAggregator" \
  --desired-state  '{"RegionLinkingMode":"ALL_REGIONS_EXCEPT_SPECIFIED","Regions":["ap-southeast-1","ap-southeast-2","ap-southeast-3","ap-southeast-4"]}'
{
    "ProgressEvent": {
        "TypeName": "AWS::SecurityHub::FindingAggregator",
        "RequestToken": "9df3b4bb-c886-4533-9f31-fcde5eff6ae8",
        "Operation": "CREATE",
        "OperationStatus": "IN_PROGRESS",
        "EventTime": "2024-07-22T16:05:36.300000-04:00"
    }
}

Get resource call

aws cloudcontrol get-resource --type-name AWS::SecurityHub::FindingAggregator --identifier "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796"
{
    "TypeName": "AWS::SecurityHub::FindingAggregator",
    "ResourceDescription": {
        "Identifier": "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796",
        "Properties": "{\"RegionLinkingMode\":\"ALL_REGIONS_EXCEPT_SPECIFIED\",\"FindingAggregationRegion\":\"us-east-1\",\"FindingAggregatorArn\":\"arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796\",\"Regions\":[\"ap-southeast-1\",\"ap-southeast-2\",\"ap-southeast-3\",\"ap-southeast-4\"]}"
    }
}

Update resource with the patch document

aws cloudcontrol update-resource --type-name "AWS::SecurityHub::FindingAggregator" \
--identifier "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796" \
--patch-document "[{\"op\":\"replace\",\"path\":\"/RegionLinkingMode\",\"value\":\"ALL_REGIONS\"}]"
{
    "ProgressEvent": {
        "TypeName": "AWS::SecurityHub::FindingAggregator",
        "Identifier": "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796",
        "RequestToken": "f77cf32a-c6e8-4f74-8f94-266532656d90",
        "Operation": "UPDATE",
        "OperationStatus": "IN_PROGRESS",
        "EventTime": "2024-07-22T16:09:47.740000-04:00",
        "ResourceModel": "{\"RegionLinkingMode\":\"ALL_REGIONS\",\"FindingAggregatorArn\":\"arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796\",\"Regions\":[\"ap-southeast-1\",\"ap-southeast-2\",\"ap-southeast-3\",\"ap-southeast-4\"]}"
    }
}

Status of the request

aws cloudcontrol get-resource-request-status --request-token "f77cf32a-c6e8-4f74-8f94-266532656d90"
{
    "ProgressEvent": {
        "TypeName": "AWS::SecurityHub::FindingAggregator",
        "Identifier": "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796",
        "RequestToken": "f77cf32a-c6e8-4f74-8f94-266532656d90",
        "Operation": "UPDATE",
        "OperationStatus": "FAILED",
        "EventTime": "2024-07-22T16:09:48.240000-04:00",
        "StatusMessage": "Regions cannot be passed as input if RegionLinkingMode is set to 'ALL_REGIONS' or 'NO_REGIONS'. (Service: AWSSecurityHub; Status Code: 400; Error Code: InvalidInputException; Request ID: e22a59a0-c17b-47a9-8d86-395604b93ef9; Proxy: null)",
        "ErrorCode": "InvalidRequest"
    }
}

quixoticmonk commented 4 months ago

Opened an internal ticket to Cloud Control API to review if the patch document is expecting the empty regions list for updating RegionLinkingMode.

quixoticmonk commented 4 months ago

Working patch document in this scenario


aws cloudcontrol update-resource --type-name "AWS::SecurityHub::FindingAggregator" \
--identifier "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796" \
--patch-document "[{\"op\":\"replace\",\"path\":\"/RegionLinkingMode\",\"value\":\"ALL_REGIONS\"},{\"op\":\"remove\",\"path\":\"Regions\"}]"
{
    "ProgressEvent": {
        "TypeName": "AWS::SecurityHub::FindingAggregator",
        "Identifier": "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796",
        "RequestToken": "cbdd01c6-dbf5-4507-bf3d-7e4eed2a6648",
        "Operation": "UPDATE",
        "OperationStatus": "IN_PROGRESS",
        "EventTime": "2024-07-22T16:53:42.198000-04:00",
        "ResourceModel": "{\"RegionLinkingMode\":\"ALL_REGIONS\",\"FindingAggregatorArn\":\"arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796\"}"
    }
}

 aws cloudcontrol get-resource-request-status --request-token "cbdd01c6-dbf5-4507-bf3d-7e4eed2a6648"
{
    "ProgressEvent": {
        "TypeName": "AWS::SecurityHub::FindingAggregator",
        "Identifier": "arn:aws:securityhub:us-east-1:############:finding-aggregator/7cc86e06-5666-6589-402c-ea91d370d796",
        "RequestToken": "cbdd01c6-dbf5-4507-bf3d-7e4eed2a6648",
        "Operation": "UPDATE",
        "OperationStatus": "SUCCESS",
        "EventTime": "2024-07-22T16:53:42.794000-04:00"
    }
}

wellsiau-aws commented 3 months ago

The problem somewhat stemmed from how AWSCC construct the patch document.

Modifying resource configuration from:

before

resource "awscc_securityhub_finding_aggregator" "example" {
  region_linking_mode = "ALL_REGIONS_EXCEPT_SPECIFIED"
  regions = [
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ap-southeast-4"
  ]
}

to after:

resource "awscc_securityhub_finding_aggregator" "example" {
  region_linking_mode = "ALL_REGIONS"
}

will yield the following plan output: (see that there's no marker to remove the regions attribute.

Terraform will perform the following actions:

  # awscc_securityhub_finding_aggregator.example will be updated in-place
  ~ resource "awscc_securityhub_finding_aggregator" "example" {
        id                         = "arn:aws:securityhub:us-east-1:204034886740:finding-aggregator/cac8beee-5eec-e65d-7663-fcc2de52bd5b"
      ~ region_linking_mode        = "ALL_REGIONS_EXCEPT_SPECIFIED" -> "ALL_REGIONS"
        # NOTICE HERE THAT THE REGIONS ATTRIBUTE UNCHANGED
    }

wellsiau-aws commented 3 months ago

Upon checking the PlanResourceChange output:

PlanResourceChange_Request_PriorState

{
  "finding_aggregation_region": "us-east-1",
  "finding_aggregator_arn": "arn:aws:securityhub:us-east-1:204034886740:finding-aggregator/cac8beee-5eec-e65d-7663-fcc2de52bd5b",
  "id": "arn:aws:securityhub:us-east-1:204034886740:finding-aggregator/cac8beee-5eec-e65d-7663-fcc2de52bd5b",
  "region_linking_mode": "ALL_REGIONS_EXCEPT_SPECIFIED",
  "regions": [
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ap-southeast-4"
  ]
}

PlanResourceChange_Request_ProposedNewState Notice how the regions are still included!

{
  "finding_aggregation_region": "us-east-1",
  "finding_aggregator_arn": "arn:aws:securityhub:us-east-1:204034886740:finding-aggregator/cac8beee-5eec-e65d-7663-fcc2de52bd5b",
  "id": "arn:aws:securityhub:us-east-1:204034886740:finding-aggregator/cac8beee-5eec-e65d-7663-fcc2de52bd5b",
  "region_linking_mode": "ALL_REGIONS",
  "regions": [
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ap-southeast-4"
  ]
}

Because the attribute regions is currently marked in AWSCC as optional and computed, the default plan modifiers will use the previous state when the configuration is empty / unknown. In this case, by not specifying the regions in the after section above, we basically trigger the regions value to unknown, which then pre-filled with the value from the previous state file.

Ref: https://github.com/hashicorp/terraform-provider-awscc/blob/v1.11.0/internal/aws/securityhub/finding_aggregator_resource_gen.go#L112-L113

wellsiau-aws commented 3 months ago

Trying to set the attribute value region to null didn't help

resource "awscc_securityhub_finding_aggregator" "example" {
  region_linking_mode = "ALL_REGIONS"
  regions = null
}

wellsiau-aws commented 3 months ago

related to #1546

one possible solution will be to remove the plan modifiers or set a default value in the schema.

hashicorp / terraform-provider-awscc