Synapse Spark Pool "Spark Configuration" Option

[olivergaudreault]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

In the Synapse Spark Pool portal there is an option for "Spark Configuration" but this isn't available in terraform. Can it be added? It is available through ARM sparkConfigProperties

New or Affected Resource(s)

• azurerm_synapse_spark_pool

Potential Terraform Configuration

N/A

References

N/A

[chrsundermann]

There is an option to configure the spark pool configuration => https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/synapse_spark_pool

spark_config {
    content  = <<EOF
spark.shuffle.spill                true
EOF
    filename = "config.txt"
  }

But that's the old way. Microsoft changed spark configuration by moving it into synapse analytics. If you create it with terraform you will get the following:

The new way must be implemented to be present at synapse analytics -> spark pool:

[fgarcia-cnb]

i have the same issue. once the params are set using the new way, the resource becomes unmanagable in terraform #20169