[Bug]: glue_data_quality_ruleset rules not supporting multi line string

[vaquarkhan]

Terraform Core Version

v1.5.4

AWS Provider Version

registry.terraform.io/hashicorp/aws v5.59.0

Affected Resource(s)

"glue_data_quality_ruleset" not supporting multiline string <<-EOF EOF

resource "aws_glue_data_quality_ruleset" "this" {
  name        = var.ruleset_name
  description =  var.description  
  ruleset   =   var.rules          #"Rules = [Completeness \"id\" between 0.4 and 0.8]"
  target_table {
    database_name = var.database_name
    table_name    = var.table_name
  }
  tags        = var.tags
}

in terraform.auto.tfvars i am using

rules         = "Rules = [Completeness \"vendorid\" between 0.4 and 0.8, Completeness \"id\" between 0.4 and 0.8,IsComplete \"vendorid\" , ColumnLength \"vendorid\" <= 91 ,Uniqueness \"vendorid\" > 0.95]"

Code is working without issue now better readability i am changing this to multi line string

rules         = <<-EOF
                  "Rules = [
                            Completeness \"vendorid\" between 0.4 and 0.8, 
                            Completeness \"id\" between 0.4 and 0.8,
                            IsComplete \"vendorid\" ,
                            ColumnLength \"vendorid\" <= 91 ,
                            Uniqueness \"vendorid\" > 0.95
                          ]"
                EOF

Now code is failed with following error message

│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "f82fff7e-8210-4b9b-b9f8-4242ebaa66e7"
│   },
│   Message_: "DataQuality rules cannot be parsed"
│ }
│
│   with module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this,
│   on ..\..\main.tf line 37, in resource "aws_glue_data_quality_ruleset" "this":
│   37: resource "aws_glue_data_quality_ruleset" "this" {

Expected Behavior

Should work without issue in both string single line vs multiline

Actual Behavior

Failed with error

│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "f82fff7e-8210-4b9b-b9f8-4242ebaa66e7"
│   },
│   Message_: "DataQuality rules cannot be parsed"
│ }
│
│   with module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this,
│   on ..\..\main.tf line 37, in resource "aws_glue_data_quality_ruleset" "this":
│   37: resource "aws_glue_data_quality_ruleset" "this" {

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

main.tf, outputs.tf ,varibles.tf

Steps to Reproduce

Use code and tfvars file

1) Create a Glue database using console 2) Create a table in database using crawler you can use simple csv and add crawler 3) After successfully run crawler you can see table and data 4) now run terraform using single line and see all working as expected now change to multiline and see errors i share above

Debug Output

│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "f82fff7e-8210-4b9b-b9f8-4242ebaa66e7"
│   },
│   Message_: "DataQuality rules cannot be parsed"
│ }
│
│   with module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this,
│   on ..\..\main.tf line 37, in resource "aws_glue_data_quality_ruleset" "this":
│   37: resource "aws_glue_data_quality_ruleset" "this" {

Panic Output

│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "f82fff7e-8210-4b9b-b9f8-4242ebaa66e7"
│   },
│   Message_: "DataQuality rules cannot be parsed"
│ }
│
│   with module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this,
│   on ..\..\main.tf line 37, in resource "aws_glue_data_quality_ruleset" "this":
│   37: resource "aws_glue_data_quality_ruleset" "this" {

Important Factoids

NA

References

NA

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @vaquarkhan 👋 Thank you for taking the time to raise this! Based on the configuration provided, I believe the issue is with your heredoc string (the reference for which may be found here). You're wrapping your string in quotes, escaping where necessary. This isn't needed in heredoc, so literally quotes will be sent with your request.

Can you try removing the quotes and testing again to verify that resolves the issue? I don't have a quick way of testing this on my end, but based on the example content you provided, it should look something like:

rules         = <<-EOF
                  Rules = [
                            Completeness "vendorid" between 0.4 and 0.8, 
                            Completeness "id" between 0.4 and 0.8,
                            IsComplete "vendorid",
                            ColumnLength "vendorid" <= 91,
                            Uniqueness "vendorid" > 0.95
                          ]
                EOF

[vaquarkhan]

Thanks @justinretzolk let me check updated rules, will keep you posted

[vaquarkhan]

[vaquarkhan]

module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this: Modifying... [id=basic_rules]
╷
│ Error: updating Glue Data Quality Ruleset (basic_rules): InvalidInputException: DataQuality rules cannot be parsed
│ {
│   RespMetadata: {
│ {
│ {
│   RespMetadata: {
│ {
│ {
│ {
│   RespMetadata: {
│ {
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "84940033-6657-4c4b-884f-36636b25acfd"
│   },
│   Message_: "DataQuality rules cannot be parsed"
│ }
│
│   with module.glue_data_quality_ruleset.aws_glue_data_quality_ruleset.this,
│   on ..\..\main.tf line 37, in resource "aws_glue_data_quality_ruleset" "this":
│   37: resource "aws_glue_data_quality_ruleset" "this" {

[vaquarkhan]

@justinretzolk r u looking any other information . please let me know if i can help you to fix issue

[justinretzolk]

Hey @vaquarkhan 👋 Thanks for the follow up here. Looks like this was a misunderstanding of the requirements on my part. The API doesn't appear to accept a multiline sting, so while you can adjust your Terraform configuration to make it a bit more readable, you'll need the end result to be a single line string with quotes escaped where needed.

I threw together a sample configuration that achieves something similar:

variable "rules" {
  type = list(string)
  default = [
    "Completeness \"vendorid\" between 0.4 and 0.8",
    "Completeness \"id\" between 0.4 and 0.8",
    "IsComplete \"vendorid\"",
    "ColumnLength \"vendorid\" <= 91",
    "Uniqueness \"vendorid\" > 0.95"
  ]
}

output "ruleset" {
  value = "Rules = [${join(", ", var.rules)}]"
}

Results:

$ terraform apply --auto-approve

Changes to Outputs:
  + ruleset = "Rules = [Completeness \"vendorid\" between 0.4 and 0.8, Completeness \"id\" between 0.4 and 0.8, IsComplete \"vendorid\", ColumnLength \"vendorid\" <= 91, Uniqueness \"vendorid\" > 0.95]"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

ruleset = "Rules = [Completeness \"vendorid\" between 0.4 and 0.8, Completeness \"id\" between 0.4 and 0.8, IsComplete \"vendorid\", ColumnLength \"vendorid\" <= 91, Uniqueness \"vendorid\" > 0.95]"

Can you give that a shot and see if it resolves your issue?

[vaquarkhan]

Thanks @justinretzolk let me validate this

[vaquarkhan]

Thanks @justinretzolk your solution works ,appreciated your help

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[vaquarkhan]

Provided solution is working as expected so closing issue.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.