azurerm_kusto_script failing to create, then saying it already exists on rerun

mojanas commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform: ~> 1.0.5
AzureRM Provider: ~> 2.83.0

Affected Resource(s)

azurerm_kusto_script

Terraform Configuration Files

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "West Europe"
}

resource "azurerm_kusto_cluster" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  sku {
    name     = "Dev(No SLA)_Standard_D11_v2"
    capacity = 1
  }
}

resource "azurerm_kusto_database" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_name        = azurerm_kusto_cluster.example.name
}

resource "azurerm_storage_account" "example" {
  name                     = "example"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "example" {
  name                  = "setup-files"
  storage_account_name  = azurerm_storage_account.example.name
  container_access_type = "private"
}

resource "azurerm_storage_blob" "example" {
  name                   = "script.txt"
  storage_account_name   = azurerm_storage_account.example.name
  storage_container_name = azurerm_storage_container.example.name
  type                   = "Block"
  source_content         = local.allCommands // All the commands put in one script
}

data "azurerm_storage_account_blob_container_sas" "example" {
  connection_string = azurerm_storage_account.example.primary_connection_string
  container_name    = azurerm_storage_container.example.name
  https_only        = true

  start  = "2017-03-21"
  expiry = "2022-03-21"

  permissions {
    read   = true
    add    = false
    create = false
    write  = true
    delete = false
    list   = true
  }
}

resource "azurerm_kusto_script" "example" {
  name                               = "example"
  database_id                        = azurerm_kusto_database.example.id
  url                                = azurerm_storage_blob.example.id
  sas_token                          = data.azurerm_storage_account_blob_container_sas.example.sas
  continue_on_errors_enabled         = false
  force_an_update_when_value_changed = "Last updated 2/24/2022"
}

locals {
   command1 = <<-EOT
      .create-merge table table1 (Column1: string, Column2: datetime)
   EOT

   command2 = <<-EOT
      .create-merge table table2 (Column1: string, Column2: datetime, Column3: dynamic)
   EOT

   command3 = <<-EOT
      .alter-merge table table2 policy retention softdelete = 30d recoverability = enabled
   EOT

   command4 = <<-EOT
      .create ifnotexists materialized-view View on table table2
      {
          table2 | summarize arg_max(Column2, *) by Column1
      }
   EOT

   command5 = <<-EOT
      .alter-materialized-view View policy retention "{}"
   EOT

    command6 = <<-EOT
      .alter materialized-view View policy caching hot = 365d
   EOT

   allCommands = "${local.command1}\n ${local.command2} \n ${local.command3}\n ${local.command4}\n ${local.command5}\n ${local.command6}"
}

Expected Behaviour

Resource (kusto script) is created and runs the commands successfully.

Actual Behaviour

First run

module.cloud.azurerm_kusto_script.adx_script["int"]: Creating...
module.cloud.azurerm_kusto_script.adx_script["int"]: Still creating... [10s elapsed]
module.cloud.azurerm_kusto_script.adx_script["int"]: Still creating... [20s elapsed]
module.cloud.azurerm_kusto_script.adx_script["int"]: Still creating... [30s elapsed]
╷
│ Error: waiting for creation of "Script: (Name \"adx_script\" / Database Name \"dbName\" / Cluster Name \"clusterName\" / Resource Group \"rgName\")": Code="Failed" Message="Internal Server Error"
│ 
│   with module.cloud.azurerm_kusto_script.adx_script["int"],
│   on ../../modules/cloud/scripts.tf line 484, in resource "azurerm_kusto_script" "adx_script":
│  484: resource "azurerm_kusto_script" "adx_script" {
│ 
╵
##[error]Bash exited with code '1'.

Seems like a transient error. When I run terraform plan again, it shows the script will be created:

Terraform will perform the following actions:

  # module.cloud.azurerm_kusto_script.adx_script["int"] will be created
  + resource "azurerm_kusto_script" "adx_script" {
      + continue_on_errors_enabled         = false
      + database_id                        = "id"
      + force_an_update_when_value_changed = "Last updated 2/23/2022"
      + id                                 = (known after apply)
      + name                               = "adx_setup"
      + sas_token                          = (sensitive)
      + url                                = "bloburl"
    }

But when I run `terraform apply again, I see this:

module.cloud.azurerm_kusto_script.adx_script["int"]: Creating...
╷
│ Error: A resource with the ID "resourceId" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_kusto_script" for more information.
│ 
│   with module.cloud.azurerm_kusto_script.adx_script["int"],
│   on ../../modules/cloud/scripts.tf line 484, in resource "azurerm_kusto_script" "adx_script":
│  484: resource "azurerm_kusto_script" "adx_script" {
│ 
╵
##[error]Bash exited with code '1'.

Sometimes, the command altering the cache policy on the materialized view will fail due to syntax error, but when I run the command in Kusto, it completes successfully.

╷
│ Error: waiting for creation of "Script: (Name \"adx_script\" / Database Name \"dbName\" / Cluster Name \"clusterName\" / Resource Group \"rgName\")": Code="ScriptFailed" Message="[BadRequest] Failed to run script 'adx_script' on database 'dbName'. Here is a list of the commands that failed: Command: .alter materialized-view View policy caching hot = 365d\r\n}  Reason: Syntax error: "
│ 
│   with module.cloud.azurerm_kusto_script.adx_script["int"],
│   on ../../modules/cloud/scripts.tf line 484, in resource "azurerm_kusto_script" "adx_script":
│  484: resource "azurerm_kusto_script" "adx_script" {
│ 
╵

Steps to Reproduce

terraform plan
terraform apply

Important Factoids

References

0000

ms-henglu commented 2 years ago

Hi @mojanas ,

Thank you for taking time to report this.

I'm not an expert of KQL, but when I paste the scripts to kusto query, it highlights command5 and seems it should be a - after .alter, after removing it, it works fine on my local. Would you please give it a try?

My config:

  command5 = <<-EOT
      .alter materialized-view View policy retention "{}"
   EOT

mojanas commented 2 years ago

Hi @ms-henglu - apologies, that was typo.

I've fixed that and also tried adding explicit dependencies on the blobs and scripts and am still hitting the "resource already exists" issue.

resource "azurerm_storage_container" "example" {
  name                  = "setup-files"
  storage_account_name  = azurerm_storage_account.example.name
  container_access_type = "private"
}

resource "azurerm_storage_blob" "example" {
  name                   = "script.txt"
  storage_account_name   = azurerm_storage_account.example.name
  storage_container_name = azurerm_storage_container.example.name
  type                   = "Block"
  source_content         = local.allCommands // All the commands put in one script

  depends_on = [
    azurerm_storage_containe.example
  ]
}

resource "azurerm_kusto_script" "example" {
  name                               = "example"
  database_id                        = azurerm_kusto_database.example.id
  url                                = azurerm_storage_blob.example.id
  sas_token                          = data.azurerm_storage_account_blob_container_sas.example.sas
  continue_on_errors_enabled         = false
  force_an_update_when_value_changed = "Last updated 2/24/2022"

  depends_on = [
    azurerm_storage_blob.example
  ]
}

mojanas commented 2 years ago

I think I figured out the issue. At some point in my script resource block I had

url = azurerm_storage_blob.example.url

instead of

url = azurerm_storage_blob.example.id

Would be nice to have some validation check for using the correct field. and/or a better error message than "Internal Server Error". I enabled debug output while running locally and I was able to see the resource create successfully but fail running the commands. My guess is ARM ran into some issue parsing the blob content, which explains why I was getting the "resource already created" error. Also would be nice to have a fix for this, maybe save the resource to terraform state after creation and then start applying the script.

ms-henglu commented 2 years ago

Hi @mojanas , But it seems url and id are set to the same value.

mojanas commented 2 years ago

Yes I noticed they were the same as well. But as soon as I changed the script resource to use blob.id it started working

mybayern1974 commented 2 years ago

@mojanas , with noticing all above contexts, would you be ok if we close this issue?

SimonG-Kontentai commented 1 year ago

I am having same issue.

resource "azurerm_kusto_script" "test" { name = "test" database_id = azurerm_kusto_database.data_explorer_database.id script_content = local.test_script continue_on_errors_enabled = false force_an_update_when_value_changed = md5(local.test_script) }

If the script contains mistake, then terraform will fail with some message -> which is correct behaviour (I suppose). BUT when I fix the mistake in the script and I run another terraform apply, it fail with a message that kusto script already exist and should be imported. How can it be created when terraform apply failed in previous run with the message that it's content is invalid?

It would be nice if in this case would be the resource marked as tainted or at least be in terraform state and something like that wouldn't be considered as new resource, but an update of existing resource, when it was already created.

holms commented 1 year ago

I have the same problem, how can this be fixed?

resource "azurerm_kusto_script" "initial_provisioning" {
  name = "initial-provisioning"
  database_id = azurerm_kusto_database.prebid.id
  script_content = file("${path.module}/scripts/initial.kusto")
  force_an_update_when_value_changed = true

  depends_on = [ azurerm_kusto_database.prebid ]
}

holms commented 1 year ago

Here's a fix I've found:

terraform import azurerm_kusto_script.provisioning /subscriptions/627949ad-9d29-4279-9a16-2aa995e3b859/resourceGroups/analytics/providers/Microsoft.Kusto/clusters/setupad/databases/prebid/scripts/provisioning

Just change database, subscription_id, cluster_name and script_name to your own.

mts-dyt commented 1 year ago

You can list and delete Kusto script with az-cli and then re-apply terraform

az kusto script list --cluster-name <CLUSTER> --database-name <DB> --resource-group <RG>
az kusto script delete --cluster-name <CLUSTER> --database-name <DB> --resource-group <RG> --name <SCRIPT_NAME>

https://learn.microsoft.com/en-us/cli/azure/kusto/script?view=azure-cli-latest

holms commented 1 year ago

You can list and delete Kusto script with az-cli and then re-apply terraform


az kusto script list --cluster-name <CLUSTER> --database-name <DB> --resource-group <RG>

az kusto script delete --cluster-name <CLUSTER> --database-name <DB> --resource-group <RG> --name <SCRIPT_NAME>

That's great to know, but it's still a bug and needs to be fixed

ebett commented 1 year ago

I am having same problem. Is there any way to remove this script using terraform just after execute it? Thanks

hashicorp / terraform-provider-azurerm