cannot update azurerm_synapse_spark_pool. complaining about invalid spark_config

fgarcia-cnb commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.0.7

AzureRM Provider Version

3.29.1

Affected Resource(s)/Data Source(s)

azurerm_synapse_spark_pool

Terraform Configuration Files

resource "azurerm_synapse_spark_pool" "synapse_spark_pool" {
  for_each = { for k, v in var.spark_pools : k => v if var.resource_count > 0 }
  name     = replace(each.key, "-", "") //name can contain only letters or numbers, must start with a letter, and be between 1 and 15 characters long

  cache_size                          = try(each.value.cache_size, var.spark_cache_size > 0 ? var.spark_node_count : null)
  compute_isolation_enabled           = try(each.value.compute_isolation_enabled, var.spark_compute_isolation_enabled)
  dynamic_executor_allocation_enabled = try(each.value.dynamic_executor_allocation_enabled, var.spark_dynamic_executor_allocation_enabled)
  node_count                          = try(each.value.node_count, var.spark_node_count > 0 ? var.spark_node_count : null)
  node_size                           = try(each.value.node_size, var.spark_node_size)
  node_size_family                    = "MemoryOptimized" //this is the only possible value
  session_level_packages_enabled      = try(each.value.session_level_packages_enabled, var.spark_session_level_packages_enabled)
  spark_events_folder                 = try(each.value.spark_events_folder, var.spark_events_folder)
  spark_log_folder                    = try(each.value.spark_log_folder, var.spark_log_folder)
  spark_version                       = try(each.value.spark_version, var.spark_version)
  synapse_workspace_id                = azurerm_synapse_workspace.synapse_workspace[each.value.workspace_index].id
  tags                                = local.tags

lifecycle {
    ignore_changes = [spark_config] //to be managed by app team pipeline
  }

  depends_on = [azurerm_synapse_workspace_key.synapse_workspace_key]
}

Debug Output/Panic Output

Error: updating Spark Pool: (Big Data Pool Name "smallPool00" / Workspace Name "td04dgc96tgi00-sw" / Resource Group "dc04-dev-cnrwm-rg2"): synapse.BigDataPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ValidationFailed" Message="Spark pool request validation failed." Details=[{"code":"InvalidSparkPoolConfigProperties","message":"Invalid content in SparkConfigProperties. Content should be space delimited key-value pairs."}]
with module.synapse.azurerm_synapse_spark_pool.synapse_spark_pool["smallPool00"]
on .terraform/modules/synapse/spark_pool.tf line 1, in resource "azurerm_synapse_spark_pool" "synapse_spark_pool":
resource "azurerm_synapse_spark_pool" "synapse_spark_pool" {

Expected Behaviour

the resource is marked to ignore spark_config parameter, but whenever an unrelated change is made, we get the error above about the sparkconfig not validating. obviously the current spark config is valid since its the current vaue for the resource.

Actual Behaviour

No response

Steps to Reproduce

No response

Important Factoids

JSON snippit of the current value of sparkConfigProperties->content in Azure portal...

"sparkConfigProperties": {
  "configurationType": "Artifact",
   "filename": "eventhub-elklm",
   "content": "{\"name\":\"eventhub-elklm\",\"properties\":{\"configs\":{\"spark.synapse.diagnostic.emitters\":\"EventHubELKLM\",\"spark.synapse.diagnostic.emitter.EventHubELKLM.type\":\"AzureEventHub\",\"spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault\":\"HIDDEN\",\"spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault.secretName\":\"eventhub-elklm\",\"spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault.linkedService\":\"kv2-managed-pe-vault\"},\"annotations\":[],\"type\":\"Microsoft.Synapse/workspaces/sparkconfigurations\",\"description\":\"\",\"notes\":\"\",\"created\":\"2023-01-25T02:16:37.4610000+05:30\",\"createdBy\":\"HIDDEN\",\"configMergeRule\":{\"admin.currentOperation.spark.synapse.diagnostic.emitters\":\"replace\",\"admin.currentOperation.spark.synapse.diagnostic.emitter.EventHubELKLM.type\":\"replace\",\"admin.currentOperation.spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault\":\"replace\",\"admin.currentOperation.spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault.secretName\":\"replace\",\"admin.currentOperation.spark.synapse.diagnostic.emitter.EventHubELKLM.secret.keyVault.linkedService\":\"replace\"}}}"
                },

References

No response

fgarcia-cnb commented 1 year ago

seems the problem is spark configs use a new method for supplying spak config params. when i let terraform deploy the spark config i see this in Azure:

unfortunately ignoring spark config changes in terraform doesnt resolve the problem. once these settings are set using the correct newer method. the resource becomes unmanagable in terraform

rajaabk commented 3 weeks ago

Any updates on this?

I am using azurerm v4.0.3 and still encountering this issue. I also attempted using a json file to match the exact configuration and the plan still fails on the spark config.

hashicorp / terraform-provider-azurerm