[Bug]: Updating AWS Service Catalog product, `terraform apply` failed, changes were applied to the state but product was untouched

[consal]

Terraform Core Version

1.4.6

AWS Provider Version

4.21.0

Affected Resource(s)

Service Catalog Product

Expected Behavior

After the terraform apply failed, the state should not be modified.

Actual Behavior

After the terraform apply failed, the state was modified with the new values.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

module "eks" {
  source = "../../modules/eks/9.0.2"

  name = "test-${var.environment}-eks"  # provisioned product name
  cluster_name = "test-${var.environment}-eks"  # actual cluster name used for nodegroups
  storage_efs_enabled = true

  external_dns_enabled = true.  # ADDED
  external_dns_hosted_zone_i_ds = module.eks-dns.outputs.oPrivateHostedZone ## ADDED
}

module "eks" is just a aws_servicecatalog_provisioned_product:

resource "aws_servicecatalog_provisioned_product" "eks" {
  name                                   = var.name
  product_id                             = local.product_id
  provisioning_artifact_id               = local.version_id

Steps to Reproduce

• Modify Terraform state so the path_id attribute of the service catalogue product is incorrect (simulate the id actually being changed because the product was moved etc.)
• Modify Terraform code and add or modify some attributes in an existing product.
• Run terraform apply and make sure it fails with Error: error updating Service Catalog Provisioned Product (pp-c5grygzt4cab2): InvalidParametersException: Invalid PathId: lpv2-pvfn5six4yb2i
• Validate in the S3 bucket Terraform state that the new values were updated
• Validate in the service catalog product that the values were not actually changed

Bonus: terraform apply -refresh-only will also say that the state is up to date

Once you change path_id to the correct one and you run apply, terraform will apply correct values in the product

Debug Output

terraform apply -var-file="vars/dev.tfvars"
aws_dynamodb_table.terraform-dynamodb: Refreshing state... [id=blob-dev-terraform-dynamodb]
module.eks-dns.aws_servicecatalog_provisioned_product.iaws_private_hosted_zone: Refreshing state... [id=pp-l6xep2amssvdw]
module.kms.aws_servicecatalog_provisioned_product.iaws_kms_support_cmk: Refreshing state... [id=pp-oiugfsfmi7vje]
module.terraform-s3.aws_servicecatalog_provisioned_product.iaws_s3: Refreshing state... [id=pp-d2yy7ruce5iv6]
module.eks.aws_servicecatalog_provisioned_product.iaws_eks: Refreshing state... [id=pp-c5grygzt4cab2]
aws_security_group_rule.allow-kafka-from-on-prem: Refreshing state... [id=sgrule-457575864]
module.eks-nodegroup-2.aws_servicecatalog_provisioned_product.iaws_eks_nodegroup: Refreshing state... [id=pp-utcsa2a2p7xww]
module.eks-nodegroup-1.aws_servicecatalog_provisioned_product.iaws_eks_nodegroup: Refreshing state... [id=pp-s6xvlyskns7c4]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks.aws_servicecatalog_provisioned_product.iaws_eks will be updated in-place
  ~ resource "aws_servicecatalog_provisioned_product" "iaws_eks" {
        id                                     = "pp-c5grygzt4cab2"
        name                                   = "blob-dev-eks"
        tags                                   = {}
        # (17 unchanged attributes hidden)

      ~ provisioning_parameters {
          ~ value              = "false" -> "true"
            # (2 unchanged attributes hidden)
        }
      ~ provisioning_parameters {
          + value              = "Z0373582Z1SFKFOI7WGJ"
            # (2 unchanged attributes hidden)
        }

        # (49 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.eks.aws_servicecatalog_provisioned_product.iaws_eks: Modifying... [id=pp-c5grygzt4cab2]
╷
│ Error: error updating Service Catalog Provisioned Product (pp-c5grygzt4cab2): InvalidParametersException: Invalid PathId: lpv2-pvfn5six4yb2i
│
│   with module.eks.aws_servicecatalog_provisioned_product.iaws_eks,
│   on ../../modules/iaws-eks/9.0.2/main.tf line 7, in resource "aws_servicecatalog_provisioned_product" "iaws_eks":
│    7: resource "aws_servicecatalog_provisioned_product" "iaws_eks" {

And then on the second run immediately after:

aws_dynamodb_table.terraform-dynamodb: Refreshing state... [id=blob-dev-terraform-dynamodb]
module.eks-dns.aws_servicecatalog_provisioned_product.iaws_private_hosted_zone: Refreshing state... [id=pp-l6xep2amssvdw]
module.kms.aws_servicecatalog_provisioned_product.iaws_kms_support_cmk: Refreshing state... [id=pp-oiugfsfmi7vje]
module.terraform-s3.aws_servicecatalog_provisioned_product.iaws_s3: Refreshing state... [id=pp-d2yy7ruce5iv6]
module.eks.aws_servicecatalog_provisioned_product.iaws_eks: Refreshing state... [id=pp-c5grygzt4cab2]
aws_security_group_rule.allow-kafka-from-on-prem: Refreshing state... [id=sgrule-457575864]
module.eks-nodegroup-2.aws_servicecatalog_provisioned_product.iaws_eks_nodegroup: Refreshing state... [id=pp-utcsa2a2p7xww]
module.eks-nodegroup-1.aws_servicecatalog_provisioned_product.iaws_eks_nodegroup: Refreshing state... [id=pp-s6xvlyskns7c4]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[atamgp]

I have the same issue. A reimport of the resource in tf state did not help

aws = {
      version = "= 4.67.0"
      source  = "hashicorp/aws"
    }

Error: updating Service Catalog Provisioned Product (pp-unnf7ojzrob64): InvalidParametersException: Invalid PathId: lpv2-idjpzoaysispo

[atamgp]

Using the aws cli we could find that the path changed for the "product" The "provisioned product" refers to a last record with the old path value.

Should the provider use the path id of the product? If so this is a bug in the provider else the question is who should update or recreate the provisioned product's last record with the new value?