[Bug]: SageMaker endpoint recreation error

[ViacheslavSubotskyiArturoAI]

Terraform Core Version

1.9.6

AWS Provider Version

5.75.0

Affected Resource(s)

aws_sagemaker_endpoint

Expected Behavior

aws_sagemaker_endpoint recreation successful

Actual Behavior

getting an error: "api error ValidationException: Cannot create already existing endpoint"

Relevant Error/Panic Output Snippet

╷
│ Error: creating SageMaker Endpoint (test-project-use1-project2-us-mme-ul-1): operation error SageMaker: CreateEndpoint, https response error StatusCode: 400, RequestID: 2ccf3f6d-492e-4743-b994-6c9b168b3ade, api error ValidationException: Cannot create already existing endpoint "arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1".
│ 
│   with module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1,
│   on ../../../../../modules/inference/sagemaker.tf line 297, in resource "aws_sagemaker_endpoint" "project2-on-demand-us-mme-ul-1":
│  297: resource "aws_sagemaker_endpoint" "project2-on-demand-us-mme-ul-1" {
│ 
╵

Terraform Configuration Files

resource "aws_sagemaker_endpoint" "project2-on-demand-us-mme-ul-1" {
  name                 = local.sagemaker_endpoint.project2-on-demand-us-mme-ul-1
  endpoint_config_name = aws_sagemaker_endpoint_configuration.project2-us-mme-ul-1.name

  depends_on = [aws_cloudwatch_log_group.sagemaker_endpoint]

  lifecycle {
    replace_triggered_by = [aws_sagemaker_endpoint_configuration.project2-us-mme-ul-1]
  }
}

resource "aws_sagemaker_endpoint_configuration" "project2-us-mme-ul-1" {
  name = "${local.prefix}-project2-us-mme-ul-1"

  production_variants {
    variant_name           = "AllTraffic"
    instance_type          = "ml.g4dn.xlarge"
    initial_instance_count = 1
    initial_variant_weight = 1

    model_name = aws_sagemaker_model.project2-us-mme-model-ul.name
  }

  lifecycle {
    replace_triggered_by = [aws_sagemaker_model.project2-us-mme-model-ul]
  }
}

resource "aws_sagemaker_model" "project2-us-mme-model-ul" {
  name               = "${local.prefix}-project2-us-mme-model-ul"
  execution_role_arn = aws_iam_role.sagemaker_role.arn

  container {
    image          = data.aws_ecr_image.project2_sagemaker_endpoint.image_uri
    mode           = "MultiModel"
    model_data_url = "s3://${local.project2_model_weights_s3_bucket_name}/"
    environment = {
      ENABLE_MULTI_MODEL = true
    }
    multi_model_config {
      model_cache_setting = "Enabled"
    }
  }

  lifecycle {
    replace_triggered_by = [terraform_data.project2-us-mme-model-ul-last-modified]
  }
}
...

Steps to Reproduce

1. Create a SageMaker endpoint using the Terraform.
2. Direct Terraform to replace the SageMaker endpoint using the -replace option.
3. Observe an error.
4. Find in the debug log the reason - Terraform decides that the endpoint is deleted while the describe API returns "EndpointStatus":"Deleting".

Debug Output

codebuild_terraform_debug.txt.zip Here is a TF debug log. It includes CodeBuild header and footer messages. Output was redacted to replace real account IDs, projects, and org names.

Panic Output

No response

Important Factoids

I may try to fix this in case someone suggest a way. I failed to find the reason in the code... The logs part that is showing the reason:

2024-11-09T07:20:46.886Z [DEBUG] module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1: applying the planned Delete change
2024-11-09T07:20:46.887Z [INFO]  provider.terraform-provider-aws_v5.75.0_x5: [INFO] Deleting SageMaker Endpoint: test-project-use1-project2-us-mme-ul-1
2024-11-09T07:20:46.888Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Request Sent: tf_mux_provider="*schema.GRPCProviderServer" http.request.header.amz_sdk_request="attempt=1; max=25" http.url=https://api.sagemaker.us-east-1.amazonaws.com/ rpc.system=aws-api tf_aws.signing_region="" tf_rpc=ApplyResourceChange
  http.request.body=
  | {"EndpointName":"test-project-use1-project2-us-mme-ul-1"}
   http.request.header.content_type=application/x-amz-json-1.1 net.peer.name=api.sagemaker.us-east-1.amazonaws.com @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 aws.region=us-east-1 http.request_content_length=55 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.6 (+https://www.terraform.io) terraform-provider-aws/5.75.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.4 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 exec-env/AWS_ECS_EC2 api/sagemaker#1.166.1" tf_req_id=2368dbbc-ee97-6a9c-5b74-704fbad7e1c7 http.request.header.amz_sdk_invocation_id=6c5f2271-9ce9-4052-8d2c-a7f1998f2430 http.request.header.x_amz_date=20241109T072046Z http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************OR3A/20241109/us-east-1/sagemaker/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" tf_provider_addr=registry.terraform.io/hashicorp/aws http.method=POST http.request.header.x_amz_security_token="*****" tf_aws.sdk=aws-sdk-go-v2 @module=aws rpc.service=SageMaker http.request.header.x_amz_target=SageMaker.DeleteEndpoint rpc.method=DeleteEndpoint tf_resource_type=aws_sagemaker_endpoint timestamp=2024-11-09T07:20:46.887Z
2024-11-09T07:20:47.042Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Response Received: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=2368dbbc-ee97-6a9c-5b74-704fbad7e1c7 tf_rpc=ApplyResourceChange http.duration=154 http.status_code=200 rpc.method=DeleteEndpoint tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" http.response.header.content_type=application/x-amz-json-1.1 tf_resource_type=aws_sagemaker_endpoint @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 @module=aws http.response.body="" http.response.header.date="Sat, 09 Nov 2024 07:20:46 GMT" http.response.header.x_amzn_requestid=392059f4-fdd1-420c-8c5e-30f6c4d3e320 rpc.service=SageMaker rpc.system=aws-api aws.region=us-east-1 tf_mux_provider="*schema.GRPCProviderServer" timestamp=2024-11-09T07:20:47.041Z
2024-11-09T07:20:47.042Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: [DEBUG] Waiting for state to become: []
2024-11-09T07:20:47.042Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Request Sent: http.request.header.x_amz_date=20241109T072047Z http.url=https://api.sagemaker.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.6 (+https://www.terraform.io) terraform-provider-aws/5.75.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.4 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 exec-env/AWS_ECS_EC2 api/sagemaker#1.166.1" http.request.header.amz_sdk_invocation_id=7715e56c-1700-4414-8c32-53ae8ef3ac0b http.request.header.content_type=application/x-amz-json-1.1 http.method=POST http.request.header.amz_sdk_request="attempt=1; max=25" net.peer.name=api.sagemaker.us-east-1.amazonaws.com rpc.service=SageMaker @module=aws tf_resource_type=aws_sagemaker_endpoint http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************OR3A/20241109/us-east-1/sagemaker/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request_content_length=55 tf_provider_addr=registry.terraform.io/hashicorp/aws http.request.header.x_amz_security_token="*****" @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 aws.region=us-east-1 rpc.system=aws-api tf_mux_provider="*schema.GRPCProviderServer" tf_req_id=2368dbbc-ee97-6a9c-5b74-704fbad7e1c7
  http.request.body=
  | {"EndpointName":"test-project-use1-project2-us-mme-ul-1"}
   rpc.method=DescribeEndpoint tf_aws.signing_region="" tf_rpc=ApplyResourceChange http.request.header.x_amz_target=SageMaker.DescribeEndpoint tf_aws.sdk=aws-sdk-go-v2 timestamp=2024-11-09T07:20:47.042Z
2024-11-09T07:20:47.123Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Response Received: rpc.service=SageMaker rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_provider_addr=registry.terraform.io/hashicorp/aws http.status_code=200 aws.region=us-east-1 http.response_content_length=854 @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 http.response.header.content_type=application/x-amz-json-1.1 http.response.header.x_amzn_requestid=0b159c95-5d7f-437f-ad70-3d99825f1b32 rpc.method=DescribeEndpoint tf_req_id=2368dbbc-ee97-6a9c-5b74-704fbad7e1c7 tf_resource_type=aws_sagemaker_endpoint tf_rpc=ApplyResourceChange http.duration=80 @module=aws
  http.response.body=
  | {"CreationTime":1.731136341809E9,"EndpointArn":"arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1","EndpointConfigName":"test-project-use1-project2-us-mme-ul-1","EndpointName":"test-project-use1-project2-us-mme-ul-1","EndpointStatus":"Deleting","LastModifiedTime":1.731136847028E9,"ProductionVariants":[{"CurrentInstanceCount":1,"CurrentWeight":1.0,"DeployedImages":[{"ResolutionTime":1.731136342785E9,"ResolvedImage":"<ACC2_ID>.dkr.ecr.us-east-1.amazonaws.com/test-project-use1-sagemaker-endpoint@sha256:de0e78f2ec81630fadabf1078fe6f185af384c618d0482cfe393d3aa56a2b141","SpecifiedImage":"<ACC2_ID>.dkr.ecr.us-east-1.amazonaws.com/test-project-use1-sagemaker-endpoint@sha256:de0e78f2ec81630fadabf1078fe6f185af384c618d0482cfe393d3aa56a2b141"}],"DesiredInstanceCount":1,"DesiredWeight":1.0,"VariantName":"AllTraffic"}]}
   http.response.header.date="Sat, 09 Nov 2024 07:20:46 GMT" tf_aws.signing_region="" tf_mux_provider="*schema.GRPCProviderServer" timestamp=2024-11-09T07:20:47.123Z
module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1: Destruction complete after 0s
2024-11-09T07:20:47.143Z [DEBUG] State storage *remote.State declined to persist a state snapshot
module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1: Creating...
2024-11-09T07:20:47.153Z [INFO]  Starting apply for module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1
2024-11-09T07:20:47.154Z [DEBUG] module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1: applying the planned Create change
2024-11-09T07:20:47.156Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Request Sent: @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 @module=aws
  http.request.body=
  | {"EndpointConfigName":"test-project-use1-project2-us-mme-ul-1","EndpointName":"test-project-use1-project2-us-mme-ul-1","Tags":[{"Key":"ManagedBy","Value":"Terraform"}]}
   http.request.header.x_amz_date=20241109T072047Z net.peer.name=api.sagemaker.us-east-1.amazonaws.com http.request.header.amz_sdk_invocation_id=64792107-08a2-483e-a1ca-1c7ec33e8249 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************OR3A/20241109/us-east-1/sagemaker/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request_content_length=164 aws.region=us-east-1 http.request.header.amz_sdk_request="attempt=1; max=25" tf_aws.sdk=aws-sdk-go-v2 tf_provider_addr=registry.terraform.io/hashicorp/aws http.request.header.x_amz_target=SageMaker.CreateEndpoint http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.6 (+https://www.terraform.io) terraform-provider-aws/5.75.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.4 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 exec-env/AWS_ECS_EC2 api/sagemaker#1.166.1" tf_req_id=f65af88c-02d4-0440-3d57-8e71fc62614e tf_resource_type=aws_sagemaker_endpoint tf_rpc=ApplyResourceChange rpc.service=SageMaker tf_aws.signing_region="" http.method=POST http.request.header.content_type=application/x-amz-json-1.1 tf_mux_provider="*schema.GRPCProviderServer" http.request.header.x_amz_security_token="*****" rpc.method=CreateEndpoint http.url=https://api.sagemaker.us-east-1.amazonaws.com/ rpc.system=aws-api timestamp=2024-11-09T07:20:47.155Z
2024-11-09T07:20:47.774Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: HTTP Response Received: http.response.header.x_amzn_requestid=2ccf3f6d-492e-4743-b994-6c9b168b3ade tf_aws.signing_region="" tf_mux_provider="*schema.GRPCProviderServer" tf_rpc=ApplyResourceChange http.response.header.content_type=application/x-amz-json-1.1 http.response_content_length=176 http.status_code=400 rpc.service=SageMaker tf_aws.sdk=aws-sdk-go-v2 tf_resource_type=aws_sagemaker_endpoint @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 @module=aws
  http.response.body=
  | {"__type":"ValidationException","message":"Cannot create already existing endpoint \"arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1\"."}
   rpc.system=aws-api tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=f65af88c-02d4-0440-3d57-8e71fc62614e aws.region=us-east-1 http.duration=618 http.response.header.date="Sat, 09 Nov 2024 07:20:47 GMT" rpc.method=CreateEndpoint timestamp=2024-11-09T07:20:47.774Z
2024-11-09T07:20:47.774Z [DEBUG] provider.terraform-provider-aws_v5.75.0_x5: request failed with unretryable error https response error StatusCode: 400, RequestID: 2ccf3f6d-492e-4743-b994-6c9b168b3ade, api error ValidationException: Cannot create already existing endpoint "arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1".: tf_rpc=ApplyResourceChange @module=aws rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_mux_provider="*schema.GRPCProviderServer" tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_sagemaker_endpoint rpc.method=CreateEndpoint aws.region=us-east-1 rpc.service=SageMaker tf_req_id=f65af88c-02d4-0440-3d57-8e71fc62614e @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.59/logging/tf_logger.go:45 timestamp=2024-11-09T07:20:47.774Z
2024-11-09T07:20:47.776Z [ERROR] provider.terraform-provider-aws_v5.75.0_x5: Response contains error diagnostic: diagnostic_summary="creating SageMaker Endpoint (test-project-use1-project2-us-mme-ul-1): operation error SageMaker: CreateEndpoint, https response error StatusCode: 400, RequestID: 2ccf3f6d-492e-4743-b994-6c9b168b3ade, api error ValidationException: Cannot create already existing endpoint \"arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1\"." tf_proto_version=5.7 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_sagemaker_endpoint @caller=github.com/hashicorp/terraform-plugin-go@v0.25.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_severity=ERROR tf_req_id=f65af88c-02d4-0440-3d57-8e71fc62614e tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_detail="" timestamp=2024-11-09T07:20:47.776Z
2024-11-09T07:20:47.776Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-11-09T07:20:47.776Z [ERROR] vertex "module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1" error: creating SageMaker Endpoint (test-project-use1-project2-us-mme-ul-1): operation error SageMaker: CreateEndpoint, https response error StatusCode: 400, RequestID: 2ccf3f6d-492e-4743-b994-6c9b168b3ade, api error ValidationException: Cannot create already existing endpoint "arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1".
2024-11-09T07:20:47.777Z [DEBUG] states/remote: state read serial is: 151; serial is: 151
2024-11-09T07:20:47.777Z [DEBUG] states/remote: state read lineage is: 3a97848f-314c-064b-0719-db8f53912078; lineage is: 3a97848f-314c-064b-0719-db8f53912078
2024-11-09T07:20:47.799Z [INFO]  backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=987ff8e0-0358-cfa6-901b-a20ea0971f50 tf_backend.s3.bucket=test-project-use1-tf-state-<ACC1_ID> tf_backend.s3.path=project_inference.tfstate
2024-11-09T07:20:47.799Z [DEBUG] backend-s3: HTTP Request Sent: aws.region=us-east-1 aws.s3.bucket=test-project-use1-tf-state-<ACC1_ID> aws.s3.key=project_inference.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" tf_backend.operation=Put tf_backend.req_id=987ff8e0-0358-cfa6-901b-a20ea0971f50 tf_backend.s3.bucket=test-project-use1-tf-state-<ACC1_ID> tf_backend.s3.path=project_inference.tfstate http.method=PUT http.url=https://test-project-use1-tf-state-<ACC1_ID>.s3.us-east-1.amazonaws.com/project_inference.tfstate?x-id=PutObject http.request.header.x_amz_security_token="*****" http.request.body="[Redacted: 376.7 KB (385,728 bytes), Type: application/json]" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************4CWN/20241109/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-security-token;x-amz-trailer, Signature=*****" http.request.header.x_amz_date=20241109T072047Z http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER http.request.header.accept_encoding=identity http.request.header.x_amz_decoded_content_length=385646 http.request.header.content_encoding=aws-chunked http.request.header.x_amz_trailer=x-amz-checksum-sha256 http.request.header.x_amz_sdk_checksum_algorithm=SHA256 http.request.header.content_type=application/json http.request.header.amz_sdk_request="attempt=1; max=5" net.peer.name=test-project-use1-tf-state-<ACC1_ID>.s3.us-east-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.6 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/linux lang/go#1.22.7 md/GOOS#linux md/GOARCH#amd64 exec-env/AWS_ECS_EC2 api/s3#1.51.4 ft/s3-transfer" http.request_content_length=385728 http.request.header.amz_sdk_invocation_id=8f0a3cd4-24a5-48e8-9819-9ff7e2df6d0f
2024-11-09T07:20:47.891Z [DEBUG] backend-s3: HTTP Response Received: aws.region=us-east-1 aws.s3.bucket=test-project-use1-tf-state-<ACC1_ID> aws.s3.key=project_inference.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" tf_backend.operation=Put tf_backend.req_id=987ff8e0-0358-cfa6-901b-a20ea0971f50 tf_backend.s3.bucket=test-project-use1-tf-state-<ACC1_ID> tf_backend.s3.path=project_inference.tfstate http.response.header.x_amz_server_side_encryption=AES256 http.response.header.etag="\"069818587d02c4c3c4e96cfb3739869e\"" http.response.header.server=AmazonS3 http.status_code=200 http.response.header.x_amz_request_id=2BWVHTGFNFQ6FJW8 http.response.header.date="Sat, 09 Nov 2024 07:20:48 GMT" http.response.header.x_amz_version_id=_W91rbp8WG.M4fwWT240Of_VVhS5z5Yn http.response.header.x_amz_id_2="vHatR5Y2o9nwUHPc17zM+k5Tw7Dw4qJg6iHZcy3G9zQdp0iOD/e1Rtmq4gsdQM4hkLgwd5dpFTI=" http.response.header.x_amz_checksum_sha256="RZD+JeMy0p/Hhu6HpiX/ajVcUpKFspjLB9Mr6fjYZA0=" http.response.body="" http.duration=91
2024-11-09T07:20:47.891Z [DEBUG] backend-s3: HTTP Request Sent: aws.dynamodb.table_names=["test-project-use1-tf-state-locks"] aws.region=us-east-1 db.system=dynamodb rpc.method=PutItem rpc.service=DynamoDB rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" tf_backend.operation=Put tf_backend.req_id=987ff8e0-0358-cfa6-901b-a20ea0971f50 tf_backend.s3.bucket=test-project-use1-tf-state-<ACC1_ID> tf_backend.s3.path=project_inference.tfstate
  http.request.body=
  | {"Item":{"Digest":{"S":"069818587d02c4c3c4e96cfb3739869e"},"LockID":{"S":"test-project-use1-tf-state-<ACC1_ID>/project_inference.tfstate-md5"}},"TableName":"test-project-use1-tf-state-locks"}
   http.method=POST http.url=https://dynamodb.us-east-1.amazonaws.com/ net.peer.name=dynamodb.us-east-1.amazonaws.com http.request.header.amz_sdk_invocation_id=94947761-6e3a-45c1-bf79-041eaa148bc5 http.request_content_length=191 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************4CWN/20241109/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.x_amz_target=DynamoDB_20120810.PutItem http.request.header.accept_encoding=identity http.request.header.x_amz_security_token="*****" http.request.header.content_type=application/x-amz-json-1.0 http.request.header.x_amz_date=20241109T072047Z http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.6 (+https://www.terraform.io) aws-sdk-go-v2/1.25.3 os/linux lang/go#1.22.7 md/GOOS#linux md/GOARCH#amd64 exec-env/AWS_ECS_EC2 api/dynamodb#1.30.4" http.request.header.amz_sdk_request="attempt=1; max=5"
2024-11-09T07:20:47.896Z [DEBUG] backend-s3: HTTP Response Received: aws.dynamodb.table_names=["test-project-use1-tf-state-locks"] aws.region=us-east-1 db.system=dynamodb rpc.method=PutItem rpc.service=DynamoDB rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" tf_backend.operation=Put tf_backend.req_id=987ff8e0-0358-cfa6-901b-a20ea0971f50 tf_backend.s3.bucket=test-project-use1-tf-state-<ACC1_ID> tf_backend.s3.path=project_inference.tfstate http.response.header.connection=keep-alive http.response.header.x_amz_crc32=2745614147
  http.response.body=
  | {}
   http.duration=4 http.status_code=200 http.response_content_length=2 http.response.header.server=Server http.response.header.content_type=application/x-amz-json-1.0 http.response.header.date="Sat, 09 Nov 2024 07:20:47 GMT" http.response.header.x_amzn_requestid=V6D17I6MMNPNNTJA6NSCMI8GH3VV4KQNSO5AEMVJF66Q9ASUAAJG
╷
│ Error: creating SageMaker Endpoint (test-project-use1-project2-us-mme-ul-1): operation error SageMaker: CreateEndpoint, https response error StatusCode: 400, RequestID: 2ccf3f6d-492e-4743-b994-6c9b168b3ade, api error ValidationException: Cannot create already existing endpoint "arn:aws:sagemaker:us-east-1:<ACC2_ID>:endpoint/test-project-use1-project2-us-mme-ul-1".
│ 
│   with module.this.aws_sagemaker_endpoint.project2-on-demand-us-mme-ul-1,
│   on ../../../../../modules/inference/sagemaker.tf line 297, in resource "aws_sagemaker_endpoint" "project2-on-demand-us-mme-ul-1":
│  297: resource "aws_sagemaker_endpoint" "project2-on-demand-us-mme-ul-1" {
│ 
╵

References

Theoretically the issue should be somewhere here https://github.com/hashicorp/terraform-provider-aws/blob/v5.75.0/internal/service/sagemaker/endpoint.go#L437

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Potentially related #40048