Cloud Control API service UpdateResource operation completion returned: waiter state transitioned to FAILED

juanforo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Terraform v1.0.11
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.70.0
+ provider registry.terraform.io/hashicorp/awscc v0.9.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/mongodb/mongodbatlas v1.0.1

Your version of Terraform is out of date! The latest version
is 1.1.2. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

awscc_appflow_flow

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "awscc_appflow_flow" "flow" {
  provider  = awscc.target
  flow_name = var.appflow_flow_name
  kms_arn   = var.kms_arn
  source_flow_config = {
    connector_profile_name = var.source_flow_config.connector_profile_name
    connector_type         = var.source_flow_config.connector_type
    source_connector_properties = {
      salesforce = {
        enable_dynamic_field_update = var.source_flow_config.enable_dynamic_field_update
        include_deleted_records     = var.source_flow_config.include_deleted_records
        object                      = var.source_flow_config.object
      }
    }
    incremental_pull_config = {}
  }
  destination_flow_config_list = [
    {
      connector_profile_name = var.destination_flow_config.connector_profile_name
      connector_type         = var.destination_flow_config.connector_type
      destination_connector_properties = {
        event_bridge = {
          error_handling_config = {
            bucket_name         = var.destination_flow_config.bucket_name
            bucket_prefix       = var.destination_flow_config.bucket_prefix
            fail_on_first_error = var.destination_flow_config.fail_on_first_error
          }
          object = "aws.partner/appflow/salesforce.com/${var.account_id}"
        }
      }
    }
  ]

  tasks = local.task_map_list

  trigger_config = {
    trigger_type = var.trigger_type
  }

  lifecycle {
    ignore_changes = [kms_arn, destination_flow_config_list, destination_flow_config_list]
  }
}

locals {
  task_map = { for index in range(length(var.tasks)) :
    "task-${index}" => {
      connector_operator = { "salesforce" = var.tasks[index].connector_operator }
      source_fields      = try(var.tasks[index].source_fields)
      destination_field  = try(var.tasks[index].destination_field, "")
      task_type          = try(var.tasks[index].task_type)
      task_properties    = try(var.tasks[index].task_properties)
    }
  }
  task_map_list = [for k, v in local.task_map : v]
}

# terraform.tfvars
This map contains more than 60 keys, only a few ones are shown here
tasks = [
  {
    source_fields = [
      "Id",
      "ReplayId",
      "ChangeEventHeader",
      "AccountId",
      "ParentId",
      "RootAssetId",
      "CreatedDate",
      "LastModifiedDate",
      "Name",
      "Status",
      "Description",
      "RecordTypeId",
      "DBA_Name__c",
      "Physical_Address_Country__c",
      "Physical_Address_Street1__c",
      "Physical_Address_Street2__c",
      "Physical_Address_City__c",
      "Physical_Address_State__c",
      "Physical_Address_Zip__c",
      "Business_Phone_Number__c",
      "Business_Email_Address__c",
      "Business_Website__c",
      "Bank_Routing_Number__c",
      "Bank_Account_Number__c",
      "Billing_Descriptor__c",
      "Customer_Service_Phone_Number__c",
      "Customer_Service_Email__c",
      "Bank_Account_DDA_Type__c",
      "Will_you_accept_Discover_Global_Network__c",
      "Discover_ID__c",
      "Will_you_accept_Amex_OptBlue__c",
      "Amex_ESA_Number__c",
      "Want_Amex_OptBlue_Marketing__c",
      "Card_Not_Present_CNP_Percentage__c",
      "Card_Present_CP_Percentage__c",
      "Credit_Card_Average_Ticket__c",
      "ACH_Average_Ticket__c",
      "Credit_Card_Annual_Volume__c",
      "Credit_Card_Monthly_Volume__c",
      "ACH_Annual_Volume__c",
      "ACH_Monthly_Volume__c",
      "Primary_Contact__c",
      "Infinicept_Merchant_App_ID__c",
      "CC_MID__c",
      "MID_Open_Date__c",
      "MID_Closed_Date__c",
      "Status_Change_Date__c",
      "ACH_Max_Dollar_Ticket__c",
      "Account_ID__c",
      "ACH_Transaction_Soft_Limit__c",
      "ACH_Transaction_Limit__c",
      "ACH_Monthly_Soft_Limit__c",
      "ACH_Is_Suspect_Enabled__c",
      "MCC__c",
      "Clearing_Account_Use_Clearing_Account__c",
      "Clearing_Account_Disburse_Release_Days__c",
      "ACH_Daily_Limit__c",
      "Primary_Contacts_Name_First__c",
      "Primary_Contacts_Name_Middle__c",
      "Primary_Contacts_Name_Last__c",
      "Primary_Contacts_Title__c",
      "Primary_Contacts_Email__c",
      "Primary_Contacts_Phone_Number__c",
      "Contract_Signed_By_Name__c",
      "Contract_Signer_IP_Address__c",
      "Contract_Signed_On__c",
      "Contract_Signer_Email__c",
      "Contract_Version__c"
    ],
    connector_operator = "PROJECTION"
    task_type          = "Filter"
    task_properties    = null
  },
  {
     source_fields = [
      "Id"
    ],
    connector_operator = "NO_OP",
    destination_field  =  "Id",
    task_type          = "Map",
    task_properties = [{
      key   = "DESTINATION_DATA_TYPE",
      value = "id"
      },
      {
        key   = "SOURCE_DATA_TYPE",
        value = "id"
      }
    ]
  },
  {
     source_fields = [
      "ReplayId"
    ],
    connector_operator = "NO_OP",
    destination_field  =  "ReplayId",
    task_type          = "Map",
    task_properties = [{
      key   = "DESTINATION_DATA_TYPE",
      value = "string"
      },
      {
        key   = "SOURCE_DATA_TYPE",
        value = "string"
      }
    ]
  },
  {
     source_fields = [
      "ChangeEventHeader"
    ],
    connector_operator = "NO_OP",
    destination_field  =  "ChangeEventHeader",
    task_type          = "Map",
    task_properties = [{
      key   = "DESTINATION_DATA_TYPE",
      value = "complexvalue"
      },
      {
        key   = "SOURCE_DATA_TYPE",
        value = "complexvalue"
      }
    ]
  },
  {
     source_fields = [
      "AccountId"
    ],
    connector_operator = "NO_OP",
    destination_field  =  "AccountId",
    task_type          = "Map",
    task_properties = [{
      key   = "DESTINATION_DATA_TYPE",
      value = "reference"
      },
      {
        key   = "SOURCE_DATA_TYPE",
        value = "reference"
      }
    ]
  },
  {
     source_fields = [
      "ParentId"
    ],
    connector_operator = "NO_OP",
    destination_field  =  "ParentId",
    task_type          = "Map",
    task_properties = [{
      key   = "DESTINATION_DATA_TYPE",
      value = "reference"
      },
      {
        key   = "SOURCE_DATA_TYPE",
        value = "reference"
      }
    ]
  },

Debug Output

https://gist.github.com/juanforo/164465fa253db65f498514d4b69a6306

Expected Behavior

Appflow flow should be updated with the new task elements

Actual Behavior

Flow being modified, but terraform gave this error

Error: AWS SDK Go Service Operation Incomplete

   with module.dev_infra.module.appflow_asset.awscc_appflow_flow.flow,
   on ../../../Modules/b360-tfm-aws-mod-appflow/main.tf line 1, in resource "awscc_appflow_flow" "flow":
    1: resource "awscc_appflow_flow" "flow" {

 Waiting for Cloud Control API service UpdateResource operation completion returned: waiter state transitioned to FAILED.
 StatusMessage: Internal Failure. ErrorCode: InternalFailure

Steps to Reproduce

terraform plan
terraform apply

Important Factoids

References

0000

juanforo commented 2 years ago

A related error is that whenever I put around 60 values in the variable tasks, I got this error MicrosoftTeams-image (5)

A workaround that I found is to create an almost empty flow, with only two or three attributes, and then put the remaining ones, run an apply, and then, you will find the error that is the name of this issue

juanforo commented 2 years ago

I added this information, just to give additional context https://aws.github.io/aws-sdk-go-v2/docs/making-requests/#advanced-waiter-configuration-overrides

AnneRey commented 2 years ago

Hello, I'm facing this same issue and is impacting our productive environments, the terraform apply will always end with an error.

Is there any workaround for this issue?

denisgaina commented 2 years ago

Also, sometimes when I try to add some new fields to a flow in AppFlow via Terraform, they are added. But sometimes not. When they are not added, as a workaround, I delete the flow from AWS Console, and run terraform apply one more time. After this workaround, all new fields are added to the flow.

MattCWheeler commented 2 years ago

I believe this is not a bug in the awscc provider, but is actually a bug in either AWS appflow or the awscc api itself. I've gotten all of the errors listed above, and I think the one below is related. The awscc api is returning incorrect status information, and I can reproduce this without terraform.

From the AWS CLI, I can run a command like aws cloudcontrol create-resource --type-name AWS::AppFlow::Flow --desired-state "..." which returns a request token. Then with that token I can poll the get-resource-request-status endpoint like so aws cloudcontrol get-resource-request-status--request-token <token> and I'll get a series of responses indicating "IN_PROGRESS" (including for tens of seconds after the new flow is showing in the AWS console) before starting to get

  "OperationStatus": "FAILED",
   "StatusMessage": "Conflict executing request: Flow with name <flowname> already exists. (Service: Appflow, Status Code: 409, Request ID: ...)",
    "ErrorCode": "AlreadyExists"

Which is nuts, because the flow is successfully created and no flow with that name has ever existed before.

The workaround that I know of is to manually untaint the resources. A subsequent plan should show if there is, in fact, any diff. In my case, there never has been so far.

I'm going to file a ticket with AWS about this.

zhujik commented 2 years ago

I have the same issue with trying to register transit gateways:

Error: AWS SDK Go Service Operation Incomplete

  with awscc_networkmanager_transit_gateway_registration.tgw_eu_north_1,
  on cloudwan.tf line 74, in resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_north_1":
  74: resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_north_1" {

Waiting for Cloud Control API service CreateResource operation completion
returned: waiter state transitioned to FAILED. StatusMessage: Incorrect
input. (Service: NetworkManager, Status Code: 400, Request ID:
2728ee2f-8a03-4c99-9f50-d07e86cbee57). ErrorCode: AlreadyExists

Error: AWS SDK Go Service Operation Incomplete

  with awscc_networkmanager_transit_gateway_registration.tgw_eu_west_1,
  on cloudwan.tf line 79, in resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_west_1":
  79: resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_west_1" {

Waiting for Cloud Control API service CreateResource operation completion
returned: waiter state transitioned to FAILED. StatusMessage: Incorrect
input. (Service: NetworkManager, Status Code: 400, Request ID:
4d5bb688-bc0c-41a6-b5b0-28eabad20b53). ErrorCode: AlreadyExists

and even subsequent plans fail:

awscc_networkmanager_transit_gateway_registration.tgw_eu_west_1: Refreshing state... [id=global-network-xxxxx|tgw-xxxxxx]
awscc_networkmanager_transit_gateway_registration.tgw_eu_north_1: Refreshing state... [id=global-network-xxxx|tgw-xxxxyy]
╷
│ Error: AWS SDK Go Service Operation Unsuccessful
│ 
│   with awscc_networkmanager_transit_gateway_registration.tgw_eu_north_1,
│   on cloudwan.tf line 74, in resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_north_1":
│   74: resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_north_1" {
│ 
│ Calling Cloud Control API service GetResource operation returned: operation
│ error CloudControl: GetResource, https response error StatusCode: 400,
│ RequestID: c1bf028d-9ced-4552-a2cc-4821c0d5288b, InvalidRequestException:
│ AWS::NetworkManager::TransitGatewayRegistration Handler returned status
│ FAILED: Incorrect input. (Service: NetworkManager, Status Code: 400,
│ Request ID: 2ca76099-e673-4ca6-b1a9-db1a016dae0c) (HandlerErrorCode:
│ InvalidRequest, RequestToken: 806fc738-c693-4230-8aa0-c0eb35ebc168)
╵
╷
│ Error: AWS SDK Go Service Operation Unsuccessful
│ 
│   with awscc_networkmanager_transit_gateway_registration.tgw_eu_west_1,
│   on cloudwan.tf line 79, in resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_west_1":
│   79: resource "awscc_networkmanager_transit_gateway_registration" "tgw_eu_west_1" {
│ 
│ Calling Cloud Control API service GetResource operation returned: operation
│ error CloudControl: GetResource, https response error StatusCode: 400,
│ RequestID: 7d516c8f-e666-48b7-affd-d91bff24a725, InvalidRequestException:
│ AWS::NetworkManager::TransitGatewayRegistration Handler returned status
│ FAILED: Incorrect input. (Service: NetworkManager, Status Code: 400,
│ Request ID: 0d788e66-02ca-41f8-8ef2-3c1c9952bade) (HandlerErrorCode:
│ InvalidRequest, RequestToken: 4de16066-a3da-4ad0-a23d-a3cc463c9107)

xsevy commented 11 months ago

Did you come up with any solutions? I have similar error but with quicksight dashboard.

hemil70 commented 2 months ago

Yes, even I am getting same error despite using same example task given in awscc appflow documentation.

Error:

╷ │ Error: AWS SDK Go Service Operation Incomplete │ │ with awscc_appflow_flow.example, │ on main.tf line 63, in resource "awscc_appflow_flow" "example": │ 63: resource "awscc_appflow_flow" "example" { │ │ Waiting for Cloud Control API service CreateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: Invalid request provided: AWS::AppFlow::FlowCreate Flow │ request failed: [Task Validation Error: You must specify a projection task or a MAP_ALL task] (Service: Appflow, Status Code: 400, Request ID: bf5be62f-71ee-4214-bdba-e0a866e72820). │ ErrorCode: InvalidRequest

resource "awscc_appflow_flow" "example" {
  flow_name = "awsccflow"
  source_flow_config = {
    connector_type         = "CustomConnector"
    api_version            = "v3"
    connector_profile_name = "githubconnector"
    source_connector_properties = {
      custom_connector = {
        entity_name = "repos/username=<my-username>"
      }
    }
  }
  destination_flow_config_list = [{
    connector_type = "S3"
    destination_connector_properties = {
      s3 = {
        bucket_name   = aws_s3_bucket.appflow_bucket.bucket
        bucket_prefix = "awscc"
      }
    }

  }]
  tasks = [{
    source_fields     = ["name"]
    destination_field = "name"
    task_type         = "Map"

    connector_operator = {
      s3 = "NO_OP"
    }

  }]
  trigger_config = {
    trigger_type = "OnDemand"
  }
}

hashicorp / terraform-provider-awscc