400 Error during destroying NetApp CVO

[bryanheo]

Hello

We deploy NetApp CVO in AWS via Terraform Enterprise using IAM user credentials with policies (https://mysupport.netapp.com/site/info/cloud-manager-policies). When we deploy the CVOs in multiple regions, it works fine but when we destroy the resources in the multiple regions, TFE returns netapp-cloudmanager_cvo_aws error in some region (us-west-2) as shown below. CVO resources are deleted in AWS even though it shows the error but due to the error, TFE state file cannot be updated.

Even we tried destroying with administrator permission but it still shows the error Could you investigate the issue and let me know the better way to find out what permissions exactly are missing?

2022-08-11T14:51:15.733Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:15 Checking existence of CVO: &schema.ResourceData{schema:map[string]*schema.Schema{"aws_encryption_kms_key_arn":(*schema.Schema)(0xc0004f5100), "aws_encryption_kms_key_id":(*schema.Schema)(0xc0004f5000), "aws_tag":(*schema.Schema)(0xc0004f6d00), "backup_volumes_to_cbs":(*schema.Schema)(0xc0004f6600), "capacity_package_name":(*schema.Schema)(0xc0004f5500), "capacity_tier":(*schema.Schema)(0xc0004f6100), "client_id":(*schema.Schema)(0xc0004f7e00), "cloud_provider_account":(*schema.Schema)(0xc0004f6500), "cluster_floating_ip":(*schema.Schema)(0xc0004f7800), "data_encryption_type":(*schema.Schema)(0xc0004f4c00), "data_floating_ip":(*schema.Schema)(0xc0004f7900), "data_floating_ip2":(*schema.Schema)(0xc0004f7a00), "ebs_volume_size":(*schema.Schema)(0xc0004f4e00), "ebs_volume_size_unit":(*schema.Schema)(0xc0004f4f00), "ebs_volume_type":(*schema.Schema)(0xc0004f4d00), "enable_compliance":(*schema.Schema)(0xc0004f6700), "enable_monitoring":(*schema.Schema)(0xc0004f6800), "failover_mode":(*schema.Schema)(0xc0004f7100), "instance_profile_name":(*schema.Schema)(0xc0004f6300), "instance_tenancy":(*schema.Schema)(0xc0004f6200), "instance_type":(*schema.Schema)(0xc0004f5700), "iops":(*schema.Schema)(0xc0004f5f00), "is_ha":(*schema.Schema)(0xc0004f6e00), "kms_key_id":(*schema.Schema)(0xc0004f6a00), "license_type":(*schema.Schema)(0xc0004f5400), "mediator_assign_public_ip":(*schema.Schema)(0xc0004f7200), "mediator_instance_profile_name":(*schema.Schema)(0xc0004f7700), "mediator_key_pair_name":(*schema.Schema)(0xc0004f7600), "mediator_subnet_id":(*schema.Schema)(0xc0004f7500), "name":(*schema.Schema)(0xc0004f4900), "node1_subnet_id":(*schema.Schema)(0xc0004f7300), "node2_subnet_id":(*schema.Schema)(0xc0004f7400), "nss_account":(*schema.Schema)(0xc0004f5d00), "ontap_version":(*schema.Schema)(0xc0004f5200), "optimized_network_utilization":(*schema.Schema)(0xc0004f6900), "platform_serial_number":(*schema.Schema)(0xc0004f5800), "platform_serial_number_node1":(*schema.Schema)(0xc0004f6f00), "platform_serial_number_node2":(*schema.Schema)(0xc0004f7000), "provided_license":(*schema.Schema)(0xc0004f5600), "region":(*schema.Schema)(0xc0004f4a00), "route_table_ids":(*schema.Schema)(0xc0004f7d00), "security_group_id":(*schema.Schema)(0xc0004f6400), "subnet_id":(*schema.Schema)(0xc0004f5900), "svm_floating_ip":(*schema.Schema)(0xc0004f7b00), "svm_name":(*schema.Schema)(0xc0004f7f00), "svm_password":(*schema.Schema)(0xc0004f5b00), "throughput":(*schema.Schema)(0xc0004f6000), "tier_level":(*schema.Schema)(0xc0004f5c00), "upgrade_ontap_version":(*schema.Schema)(0xc0004fc000), "use_latest_version":(*schema.Schema)(0xc0004f5300), "vpc_id":(*schema.Schema)(0xc0004f5a00), "workspace_id":(*schema.Schema)(0xc0004f4b00), "writing_speed_state":(*schema.Schema)(0xc0004f5e00)}, config:(*terraform.ResourceConfig)(nil), state:(*terraform.InstanceState)(0xc0004ce8c0), diff:(*terraform.InstanceDiff)(nil), meta:map[string]interface {}(nil), timeouts:(*schema.ResourceTimeout)(0xc000208600), providerMeta:cty.Value{ty:cty.Type{typeImpl:cty.typeObject{typeImplSigil:cty.typeImplSigil{}, AttrTypes:map[string]cty.Type{}}}, v:interface {}(nil)}, multiReader:(*schema.MultiLevelFieldReader)(nil), setWriter:(*schema.MapFieldWriter)(nil), newState:(*terraform.InstanceState)(nil), partial:false, partialMap:map[string]struct {}(nil), once:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, isNew:false, panicOnError:false}
2022-08-11T14:51:15.733Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:15 getCVOAWS
2022-08-11T14:51:15.734Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:15 getAccessToken
2022-08-11T14:51:17.705Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 Reading CVO: &schema.ResourceData{schema:map[string]*schema.Schema{"aws_encryption_kms_key_arn":(*schema.Schema)(0xc0004f5100), "aws_encryption_kms_key_id":(*schema.Schema)(0xc0004f5000), "aws_tag":(*schema.Schema)(0xc0004f6d00), "backup_volumes_to_cbs":(*schema.Schema)(0xc0004f6600), "capacity_package_name":(*schema.Schema)(0xc0004f5500), "capacity_tier":(*schema.Schema)(0xc0004f6100), "client_id":(*schema.Schema)(0xc0004f7e00), "cloud_provider_account":(*schema.Schema)(0xc0004f6500), "cluster_floating_ip":(*schema.Schema)(0xc0004f7800), "data_encryption_type":(*schema.Schema)(0xc0004f4c00), "data_floating_ip":(*schema.Schema)(0xc0004f7900), "data_floating_ip2":(*schema.Schema)(0xc0004f7a00), "ebs_volume_size":(*schema.Schema)(0xc0004f4e00), "ebs_volume_size_unit":(*schema.Schema)(0xc0004f4f00), "ebs_volume_type":(*schema.Schema)(0xc0004f4d00), "enable_compliance":(*schema.Schema)(0xc0004f6700), "enable_monitoring":(*schema.Schema)(0xc0004f6800), "failover_mode":(*schema.Schema)(0xc0004f7100), "instance_profile_name":(*schema.Schema)(0xc0004f6300), "instance_tenancy":(*schema.Schema)(0xc0004f6200), "instance_type":(*schema.Schema)(0xc0004f5700), "iops":(*schema.Schema)(0xc0004f5f00), "is_ha":(*schema.Schema)(0xc0004f6e00), "kms_key_id":(*schema.Schema)(0xc0004f6a00), "license_type":(*schema.Schema)(0xc0004f5400), "mediator_assign_public_ip":(*schema.Schema)(0xc0004f7200), "mediator_instance_profile_name":(*schema.Schema)(0xc0004f7700), "mediator_key_pair_name":(*schema.Schema)(0xc0004f7600), "mediator_subnet_id":(*schema.Schema)(0xc0004f7500), "name":(*schema.Schema)(0xc0004f4900), "node1_subnet_id":(*schema.Schema)(0xc0004f7300), "node2_subnet_id":(*schema.Schema)(0xc0004f7400), "nss_account":(*schema.Schema)(0xc0004f5d00), "ontap_version":(*schema.Schema)(0xc0004f5200), "optimized_network_utilization":(*schema.Schema)(0xc0004f6900), "platform_serial_number":(*schema.Schema)(0xc0004f5800), "platform_serial_number_node1":(*schema.Schema)(0xc0004f6f00), "platform_serial_number_node2":(*schema.Schema)(0xc0004f7000), "provided_license":(*schema.Schema)(0xc0004f5600), "region":(*schema.Schema)(0xc0004f4a00), "route_table_ids":(*schema.Schema)(0xc0004f7d00), "security_group_id":(*schema.Schema)(0xc0004f6400), "subnet_id":(*schema.Schema)(0xc0004f5900), "svm_floating_ip":(*schema.Schema)(0xc0004f7b00), "svm_name":(*schema.Schema)(0xc0004f7f00), "svm_password":(*schema.Schema)(0xc0004f5b00), "throughput":(*schema.Schema)(0xc0004f6000), "tier_level":(*schema.Schema)(0xc0004f5c00), "upgrade_ontap_version":(*schema.Schema)(0xc0004fc000), "use_latest_version":(*schema.Schema)(0xc0004f5300), "vpc_id":(*schema.Schema)(0xc0004f5a00), "workspace_id":(*schema.Schema)(0xc0004f4b00), "writing_speed_state":(*schema.Schema)(0xc0004f5e00)}, config:(*terraform.ResourceConfig)(nil), state:(*terraform.InstanceState)(0xc0004ce8c0), diff:(*terraform.InstanceDiff)(nil), meta:map[string]interface {}(nil), timeouts:(*schema.ResourceTimeout)(0xc000208600), providerMeta:cty.Value{ty:cty.Type{typeImpl:cty.typeObject{typeImplSigil:cty.typeImplSigil{}, AttrTypes:map[string]cty.Type{}}}, v:interface {}(nil)}, multiReader:(*schema.MultiLevelFieldReader)(nil), setWriter:(*schema.MapFieldWriter)(nil), newState:(*terraform.InstanceState)(nil), partial:false, partialMap:map[string]struct {}(nil), once:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, isNew:false, panicOnError:false}
2022-08-11T14:51:17.705Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 getCVOAWSByID
2022-08-11T14:51:17.821Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 getWorkingEnvironmentInfo request failed: {"message":"You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies","causeMessage":"SimplicatorAuthenticationException: You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies"}
2022-08-11T14:51:17.821Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 apiResponseChecker error code: 400, message: {"message":"You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies","causeMessage":"SimplicatorAuthenticationException: You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies"}
2022-08-11T14:51:17.821Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 Cannot get working environment information.
2022-08-11T14:51:17.821Z [DEBUG] provider.terraform-provider-netapp-cloudmanager_v22.8.1: 2022/08/11 14:51:17 Error reading cvo aws
2022-08-11T14:51:17.822Z [INFO]  backend/local: plan operation completed
╷
│ Error: code: 400, message: {"message":"You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies","causeMessage":"SimplicatorAuthenticationException: You don't have permission to perform this action. For more information please refer to the OnCommand Cloud Manager policies documentation at https://mysupport.netapp.com/site/info/cloud-manager-policies"}
│ 
│   with module.usw2.module.cvo.netapp-cloudmanager_cvo_aws.this,
│   on .terraform/modules/usw2/modules/cvo/cvo.tf line 1, in resource "netapp-cloudmanager_cvo_aws" "this":
│    1: resource "netapp-cloudmanager_cvo_aws" "this" {

Regards Moon

[bryanheo]

I think this could be TF dependancy issue