Customer managed key for storage account not working : "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for .network_rules[0].ip_rules:

rranjan82 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.6

AzureRM Provider Version

3.60.0

Affected Resource(s)/Data Source(s)

azurerm_storage

Terraform Configuration Files

module "storage_account_syn" {
  source                   = "../../templates/modules/storage/storage_account/v1"
  name                     = "${local.restricted_base_name_syn}${var.syn_environment}sa"
  location                 = module.resource_group_syn.location
  resource_group_name      = module.resource_group_syn.name
  account_tier             = "Standard"
  account_replication_type = "RAGRS"
  allow_nested_items_to_be_public = true
  enable_https_traffic_only = true
  min_tls_version           = "TLS1_2"
  public_network_access_enabled = false
  shared_access_key_enabled = false
  default_to_oauth_authentication = true
  environment_variables    = module.environment_variables.values
  role_assignments         = var.syn_storage_role_assignments

  network_rules = {
    default_action             = "Deny"                 # (Optional) Specifies the default action of Allow or Deny when no other rules match.  Valid options are Deny or Allow.  Defaults to Deny.
    bypass                     = ["AzureServices"]      # (Optional) A set specifying the type(s) of traffic that will bypass this rule.  Can be one or more of: Logging, Metrics, AzureServices and None.
                                                        # If unspecified, a default of AzureServices will be used.
    ip_rules                   = var.syn_ip_access_list     # (Optional ) A list of public IPs/IP ranges in CIDR format (IPv4 only).  Private IP address ranges (ref: RFC 1918) are not allowed.
                                                        # At the very least, the Manulife Desktop CIDR range (165.225.208.0/23, zscaler *Toronto III location only*) will be used, to help
                                                        # prevent access post-deployment Storage Account access issues in Azure Portal, as well as permissions issues during pipeline operations.
                                                        # To find the correct CIDR for your infrastructure, please visit the zscaler Cloud Enforcement Node Ranges site at https://config.zscaler.com/zscaler.net/cenr.
    virtual_network_subnet_ids = var.syn_subnet_access_list # (Optional ) A list of resource IDs for subnets.   If unspecified, the Azure default will be used.

    manulife_services = ["CanadianJenkins", "zscaler_canada", "zscaler_us"]

  }

  encryption_type = "CustomManaged" # "MicrosoftManaged"
  key_vault_id    = module.key_vault_syn.id
  key_name        = "${local.restricted_base_name_syn}${var.syn_environment}akv${var.syn_keyvault_iteration}"
  key_version     = module.key_vault_syn.key_vaul_keys_ids["syn-cmk-sandbox"]
  identity = {
    type = "SystemAssigned"
  }

Debug Output/Panic Output

17:46:03  module.synapse_analytics_workspace.module.private_endpoints["synapse-dev-pe"].azurerm_private_endpoint.endpoint: Creation complete after 3m19s [id=/subscriptions/7b97a7f5-9c8b-4161-b382-dc3951aab88a/resourceGroups/ihub-poc-syn-sandbox-rg/providers/Microsoft.Network/privateEndpoints/synapse-dev-pe]
17:46:03  
17:46:03  Error: Provider produced inconsistent final plan
17:46:03  
17:46:03  When expanding the plan for
17:46:03  module.storage_account_syn.azurerm_storage_account.storage_account to include
17:46:03  new values learned so far during apply, provider
17:46:03  "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
17:46:03  .network_rules[0].ip_rules: actual set element
17:46:03  cty.StringVal("170.85.26.0/23") does not correlate with any element in plan.
17:46:03  
17:46:03  This is a bug in the provider, which should be reported in the provider's own
17:46:03  issue tracker.
17:46:03  
17:46:03  Error: Provider produced inconsistent final plan
17:46:03  
17:46:03  When expanding the plan for
17:46:03  module.storage_account_syn.azurerm_storage_account.storage_account to include
17:46:03  new values learned so far during apply, provider
17:46:03  "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
17:46:03  .network_rules[0].ip_rules: length changed from 90 to 91.
17:46:03  
17:46:03  This is a bug in the provider, which should be reported in the provider's own
17:46:03  issue tracker.
[Pipeline] readFile
[Pipeline] ansiColor

Expected Behaviour

Expectation is that, customer manage key should get enabled for storage account.

Actual Behaviour

terraform plan is working fine. terraform apply is failing.

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

magodo commented 1 year ago

@rranjan82 From the error message:

17:46:03 Error: Provider produced inconsistent final plan 17:46:03
17:46:03 When expanding the plan for 17:46:03 module.storage_account_syn.azurerm_storage_account.storage_account to include 17:46:03 new values learned so far during apply, provider 17:46:03 "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for 17:46:03 .network_rules[0].ip_rules: length changed from 90 to 91.

It appears that the ip_rules you've set in the module input by var.syn_ip_access_list is of length 90, while after the apply, it reads 91 ip_rules from Azure. It mostly because you are mixing azurerm_storage_account_network_rules and the network_rules in azurem_storage_account. See the highlight at the begining of the document of the azurerm_storage_account_network_rules:

Network Rules can be defined either directly on the azurerm_storage_account resource, or using the azurerm_storage_account_network_rules resource - but the two cannot be used together. Spurious changes will occur if both are used against the same Storage Account.

And:

Only one azurerm_storage_account_network_rules can be tied to an azurerm_storage_account. Spurious changes will occur if more than azurerm_storage_account_network_rules is tied to the same azurerm_storage_account.

rranjan82 commented 1 year ago

Thanks!. I have updated the code to have one IP rule.

Now it is atleast trying to enable customer key but after 7 minutes, it fails with below error. Any idea why?

02:27:17 module.storage_account_syn1.azurerm_storage_account_customer_managed_key.storage_encryption[0]: Still creating... [7m0s elapsed] 02:27:25 module.storage_account_syn1.azurerm_storage_account_customer_managed_key.storage_encryption[0]: Still creating... [7m10s elapsed] 02:27:35 module.storage_account_syn1.azurerm_storage_account_customer_managed_key.storage_encryption[0]: Still creating... [7m20s elapsed] 02:27:45 module.storage_account_syn1.azurerm_storage_account_customer_managed_key.storage_encryption[0]: Still creating... [7m30s elapsed] 02:27:49
02:27:49 Error: updating Customer Managed Key for Storage Account "ihubpocsynsandboxsa1" (Resource Group "ihub-poc-syn-sandbox-rg"): storage.AccountsClient#Update: Failure responding to request: StatusCode=500 -- Original Error: autorest/azure: Service returned an error. Status=500 Code="GatewayAuthenticationFailed" Message="Gateway authentication failed for 'Microsoft.Storage'. Diagnostic information: timestamp '20230627T062347Z', tracking id '7ce8b20f-2c48-48db-9573-3fc5a1818520', request correlation id '07d6fc91-61aa-5076-8d91-df77691ec442'." 02:27:49
02:27:49 with module.storage_account_syn1.azurerm_storage_account_customer_managed_key.storage_encryption[0], 02:27:49 on ../../templates/modules/storage/storage_account/v1/storage_account_main.tf line 497, in resource "azurerm_storage_account_customer_managed_key" "storage_encryption": 02:27:49 497: resource "azurerm_storage_account_customer_managed_key" "storage_encryption" {

hashicorp / terraform-provider-azurerm