azurerm_data_factory_dataset_delimited_text required rowDelimiter field prevents auto-detection

[ScoGroMSFT]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.7

• provider registry.terraform.io/hashicorp/azurerm v2.49.0

Affected Resource(s)

azurerm_data_factory_dataset_delimited_text

Terraform Configuration Files

This is nearly verbatim what is in the example doc, with the exception that it points to an Azure blob location, rather than a raw http location.

resource "azurerm_data_factory_dataset_delimited_text" "csvdataset" {
  name                = "csvdataset"
  resource_group_name = azurerm_resource_group.rg.name
  data_factory_name   = azurerm_data_factory.datafactory.name
  linked_service_name = azurerm_data_factory_linked_service_azure_blob_storage.csvlinkedservice.name

  azure_blob_storage_location {
    container = azurerm_storage_blob.csvblob.storage_container_name
    path     = "/"
    filename = azurerm_storage_blob.csvblob.name
  }

  column_delimiter    = ","
  row_delimiter       = "NEW"
  encoding            = "UTF-8"
  quote_character     = "\""
  escape_character    = "f"
  first_row_as_header = true
  null_value          = "NULL"
}

Debug Output

Debug output is included here but the issue is present when using the example exactly as presented in the doc: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_dataset_delimited_text

https://gist.github.com/ScoGroMSFT/a4404a55273ae813b8fa2044eb37e8c7

Panic Output

N/A

Expected Behaviour

I should be able to omit the row delimiter to use the default detection option (which supports \r, \n, or \r\n)

Actual Behaviour

The row delimiter is required. The example gives the text "NEW" which would only be valid if that string was used to terminate lines.

Steps to Reproduce

Use the example TF to create a dataset resource. Observe that you must specify a row delimiter.

1. terraform apply

Important Factoids

I am using a workaround that I published here: https://github.com/ScoGroMSFT/examples/blob/dfe62e813d8b568eed7fdeac5485e62bee5ed050/adf-terraform/data_factory.tf#L40

It uses the RM to read the resource definition,, strip the rowDelimiter property, and write it back to RM.

References

N/A

• 0000

[favoretti]

Hi @ScoGroMSFT, thank you for reporting this. Looking at the code - row_delimiter is an optional parameter, so you should be able to omit it from your configuration:

// Delimited Text Specific Field
"row_delimiter": {
    Type:         schema.TypeString,
    Optional:     true,
    ValidateFunc: validation.StringIsNotEmpty,
},

[ScoGroMSFT]

It looks like there are a few things going on:

1 - there's a doc bug that states this is required: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_dataset_delimited_text

2 - when I remove that field from my TF, I end up with the value being set to No Delimiter, rather than the default of auto-detect.

Here's the trace from running #2: https://gist.github.com/ScoGroMSFT/99514dcb1e617e75192a3a2b83b06492

[njuCZ]

@ScoGroMSFT I had an investigation of this issue and found if pass nil for rowDelimiter, the backend service will treat it as default of auto-detect, if pass "" for rowDelimiter, the backend service will treat it as no delimiter.

However there is a bug for terraform-sdk-plugin, we could not differentiate between nil and empty string. So this issue is currently blocked

[marcobustillo]

Hi @njuCZ, I have currently hit this roadblock. Do you happen to have any updates on this? How should I proceed with default values on the row_delimeter attribute? I am now using the latest version of azurerm which this issue is still not yet fixed