search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.83k stars 9.19k forks source link

[Bug]: aws_quicksight_template has mismatches with the AWS API #34703

Open joseph-wortmann opened 11 months ago

joseph-wortmann commented 11 months ago

Terraform Core Version

1.5.4

AWS Provider Version

5.28.0

Affected Resource(s)

aws_quicksight_template

I suspect that It also affects the aws_quicksight_dashboard and aws_quicksight_analysis when the definition field is used.

Expected Behavior

Took the output template definition from a QuickSight template and converted It fairly directly to TF. Given that the provider references the API, I expected this to work.

Actual Behavior

This exposed several bugs in the provider for the definition field on the template.

I suspect that there may be more mismatches given the way this resource is written.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_quicksight_template" "temp_jw" { template_id = "temp_jw" name = "temp_jw" version_description = "Initial version" definition { data_set_configurations { placeholder = "Bordereau" data_set_schema { column_schema_list { name = "loss_expense_payment" data_type = "DECIMAL" } column_schema_list { name = "loss_payment" data_type = "DECIMAL" } column_schema_list { name = "loss_recovery_expense_payment" data_type = "DECIMAL" } column_schema_list { name = "losstype" data_type = "STRING" } column_schema_list { name = "payeename" data_type = "STRING" } column_schema_list { name = "id" data_type = "STRING" } column_schema_list { name = "accountstate" data_type = "STRING" } column_schema_list { name = "loss_recovery_payment" data_type = "DECIMAL" } column_schema_list { name = "createtime" data_type = "DATETIME" } }

}
sheets {
  sheet_id = "a2cfe993-c3ce-4949-b238-5cfbb38eea3e"
  name     = "Sheet 1"
  visuals {
    table_visual {
      visual_id = "d101b8a3-da96-4bfc-8540-d7b03a24cf5f"
      title {
        visibility = "VISIBLE"
        format_text {
          rich_text = <<EOT
Detail by Insured State

EOT } } subtitle { visibility = "VISIBLE" } chart_configuration { field_wells { table_aggregated_field_wells { group_by { categorical_dimension_field { field_id = "bordereau.accountstate.0.1701432581418" column { data_set_identifier = "Bordereau" column_name = "accountstate" } } } group_by { categorical_dimension_field { field_id = "bordereau.id.1.1701432585901" column { data_set_identifier = "Bordereau" column_name = "id" } } } group_by { categorical_dimension_field { field_id = "bordereau.payeename.5.1701432618067" column { data_set_identifier = "Bordereau" column_name = "payeename" } } } group_by { categorical_dimension_field { field_id = "bordereau.losstype.7.1701432895948" column { data_set_identifier = "Bordereau" column_name = "losstype" } } } group_by { date_dimension_field { field_id = "bordereau.createtime.7.1701433024862" column { data_set_identifier = "Bordereau" column_name = "createtime" } format_configuration { date_time_format = "MM-DD" } } } values { numerical_measure_field { field_id = "bordereau.loss_payment.2.1701432590132" column { data_set_identifier = "Bordereau" column_name = "loss_payment" } aggregation_function { simple_numerical_aggregation = "SUM" } format_configuration { format_configuration { currency_display_format_configuration { separator_configuration { decimal_separator = "DOT" thousands_separator { symbol = "COMMA" visibility = "VISIBLE" } } symbol = "USD" decimal_places_configuration { decimal_places = 2 } negative_value_configuration { display_mode = "POSITIVE" } } } } } } values { numerical_measure_field { field_id = "bordereau.loss_expense_payment.3.1701432591176" column { data_set_identifier = "Bordereau" column_name = "loss_expense_payment" } aggregation_function { simple_numerical_aggregation = "SUM" } format_configuration { format_configuration { currency_display_format_configuration { separator_configuration { decimal_separator = "DOT" thousands_separator { symbol = "COMMA" visibility = "VISIBLE" } } symbol = "USD" decimal_places_configuration { decimal_places = 2 } negative_value_configuration { display_mode = "POSITIVE" } } } } } } values { numerical_measure_field { field_id = "bordereau.loss_recovery_payment.8.1701434357051" column { data_set_identifier = "Bordereau" column_name = "loss_recovery_payment" } aggregation_function { simple_numerical_aggregation = "SUM" } format_configuration { format_configuration { currency_display_format_configuration { separator_configuration { decimal_separator = "DOT" thousands_separator { symbol = "COMMA" visibility = "VISIBLE" } } symbol = "USD" decimal_places_configuration { decimal_places = 2 } negative_value_configuration { display_mode = "POSITIVE" } } } } } } values { numerical_measure_field { field_id = "bordereau.loss_recovery_expense_payment.9.1701434359184" column { data_set_identifier = "Bordereau" column_name = "loss_recovery_expense_payment" } aggregation_function { simple_numerical_aggregation = "SUM" } format_configuration { format_configuration { currency_display_format_configuration { separator_configuration { decimal_separator = "DOT" thousands_separator { symbol = "COMMA" visibility = "VISIBLE" } } symbol = "USD" decimal_places_configuration { decimal_places = 2 } negative_value_configuration { display_mode = "POSITIVE" } } } } } } values { numerical_measure_field { field_id = "f68bece9-d748-4b51-b473-34a34d3bed47.9.1701436004524" column { data_set_identifier = "Bordereau" column_name = "loss_total" } aggregation_function { simple_numerical_aggregation = "SUM" } format_configuration { format_configuration { currency_display_format_configuration { separator_configuration { decimal_separator = "DOT" thousands_separator { symbol = "COMMA" visibility = "VISIBLE" } } symbol = "USD" decimal_places_configuration { decimal_places = 2 } negative_value_configuration { display_mode = "POSITIVE" } } } } } } } } sort_configuration { row_sort { field_sort { field_id = "bordereau.losstype.7.1701432895948" direction = "ASC" } } } table_options { header_style { text_wrap = "WRAP" height = 25 } cell_style { height = 25 } } total_options { totals_visibility = "VISIBLE" placement = "END" } field_options { selected_field_options { field_id = "bordereau.accountstate.0.1701432581418" width = "55px" custom_label = "State" } selected_field_options { field_id = "bordereau.id.1.1701432585901" custom_label = "Claim" } selected_field_options { field_id = "bordereau.payeename.5.1701432618067" width = "243px" custom_label = "Insured" } selected_field_options { field_id = "bordereau.losstype.7.1701432895948" width = "83px" custom_label = "Type" } selected_field_options { field_id = "bordereau.createtime.7.1701433024862" width = "54px" custom_label = "Paid" } selected_field_options { field_id = "bordereau.loss_payment.2.1701432590132" width = "140px" custom_label = "Loss" } selected_field_options { field_id = "bordereau.loss_expense_payment.3.1701432591176" width = "145px" custom_label = "Loss Expense" } selected_field_options { field_id = "bordereau.loss_recovery_payment.8.1701434357051" width = "138px" custom_label = "Recovery" } selected_field_options { field_id = "bordereau.loss_recovery_expense_payment.9.1701434359184" width = "145px" custom_label = "Recovery Expense" } selected_field_options { field_id = "f68bece9-d748-4b51-b473-34a34d3bed47.9.1701436004524" width = "167px" custom_label = "Total" }

        }
        table_inline_visualizations {
          data_bars {
            field_id       = "bordereau.loss_payment.2.1701432590132"
            positive_color = "#2CAD00"
            negative_color = "#DE3B00"
          }
        }
        table_inline_visualizations {
          data_bars {
            field_id       = "bordereau.loss_expense_payment.3.1701432591176"
            positive_color = "#2CAD00"
            negative_color = "#DE3B00"
          }
        }
        table_inline_visualizations {
          data_bars {
            field_id       = "bordereau.loss_recovery_payment.8.1701434357051"
            positive_color = "#2CAD00"
            negative_color = "#DE3B00"
          }
        }
        table_inline_visualizations {
          data_bars {
            field_id       = "bordereau.loss_recovery_expense_payment.9.1701434359184"
            positive_color = "#2CAD00"
            negative_color = "#DE3B00"
          }
        }
        table_inline_visualizations {
          data_bars {
            field_id       = "f68bece9-d748-4b51-b473-34a34d3bed47.9.1701436004524"
            positive_color = "#2CAD00"
            negative_color = "#DE3B00"
          }
        }
      }
      conditional_formatting {

      }

    }
  }
  visuals {
    bar_chart_visual {
      visual_id = "53fb070b-f751-425d-83c9-c688916cb583"
      title {
        visibility = "VISIBLE"
        format_text {
          rich_text = <<EOT
By Loss Type and Insured State

EOT } } subtitle { visibility = "VISIBLE" } chart_configuration { field_wells { bar_chart_aggregated_field_wells { category { categorical_dimension_field { field_id = "bordereau.losstype.0.1701436504877" column { data_set_identifier = "Bordereau" column_name = "losstype" } } } values { numerical_measure_field { field_id = "f68bece9-d748-4b51-b473-34a34d3bed47.1.1701436506542" column { data_set_identifier = "Bordereau" column_name = "loss_total" } aggregation_function { simple_numerical_aggregation = "SUM" } } } colors { categorical_dimension_field { field_id = "bordereau.accountstate.2.1701436508904" column { data_set_identifier = "Bordereau" column_name = "accountstate" } } } } } sort_configuration { category_sort { field_sort { field_id = "bordereau.losstype.0.1701436504877" direction = "DESC" } } category_items_limit { other_categories = "INCLUDE" } color_items_limit { other_categories = "INCLUDE" } small_multiples_limit_configuration { other_categories = "INCLUDE" } } orientation = "HORIZONTAL" bars_arrangement = "STACKED" category_axis { tick_label_options { label_options { font_configuration { font_size { relative = "LARGE" } } } } scrollbar_options { visible_range { percent_range {

                to = 100.0
              }
            }
          }
        }
        category_label_options {
          visibility = "HIDDEN"
        }
        legend {
          position = "BOTTOM"
        }
        data_labels {
          visibility = "VISIBLE"
          label_font_configuration {
            font_size {
              relative = "LARGE"
            }
          }
          overlap           = "DISABLE_OVERLAP"
          totals_visibility = "VISIBLE"
        }
        tooltip {
          tooltip_visibility    = "VISIBLE"
          selected_tooltip_type = "DETAILED"
          field_based_tooltip {
            aggregation_visibility = "HIDDEN"
            tooltip_title_type     = "PRIMARY_VALUE"
            tooltip_fields {
              field_tooltip_item {
                field_id   = "bordereau.losstype.0.1701436504877"
                visibility = "VISIBLE"
              }
            }
            tooltip_fields {
              field_tooltip_item {
                field_id   = "f68bece9-d748-4b51-b473-34a34d3bed47.1.1701436506542"
                visibility = "VISIBLE"
              }
            }
            tooltip_fields {
              field_tooltip_item {
                field_id   = "bordereau.accountstate.2.1701436508904"
                visibility = "VISIBLE"
              }
            }
          }
        }
      }

    }
  }
  text_boxes {
    sheet_text_box_id = "fd9ed472-390b-4ea2-97f9-fd6107f1f2af"
    content           = <<EOT
Bordereau Report By State

EOT } text_boxes { sheet_text_box_id = "806a27f2-958c-4af7-b5a7-e7aef9576dcc" content = <<EOT

${"$"}{StartDate}  to  ${"$"}{EndDate}

EOT } layouts { configuration { section_based_layout { header_sections { section_id = "92440f28-ee2b-4000-9f98-3f29d6d1b6dd" layout { free_form_layout { elements { element_id = "fd9ed472-390b-4ea2-97f9-fd6107f1f2af" element_type = "TEXT_BOX" x_axis_location = "0px" y_axis_location = "0px" width = "1248px" height = "48px" visibility = "VISIBLE" } } } style { height = "48px" } } body_sections { section_id = "a432e05c-2d89-4858-a1d4-b86a875b7af1" content { layout { free_form_layout { elements { element_id = "53fb070b-f751-425d-83c9-c688916cb583" element_type = "VISUAL" x_axis_location = "0px" y_axis_location = "0px" width = "1248px" height = "592px" visibility = "VISIBLE" } } } } } body_sections { section_id = "19f37707-87c5-44c8-8f68-d17f92231ba1" content { layout { free_form_layout { elements { element_id = "d101b8a3-da96-4bfc-8540-d7b03a24cf5f" element_type = "VISUAL" x_axis_location = "0px" y_axis_location = "0px" width = "1248px" height = "224px" visibility = "VISIBLE" } } } } } footer_sections { section_id = "4d1e7c90-d4ac-4255-b598-d18c7e4003c6" layout { free_form_layout { elements { element_id = "806a27f2-958c-4af7-b5a7-e7aef9576dcc" element_type = "TEXT_BOX" x_axis_location = "0px" y_axis_location = "0px" width = "1248px" height = "48px" visibility = "VISIBLE" } } } style { height = "48px" } } canvas_size_options { paper_canvas_size_options { paper_size = "US_LEGAL" paper_orientation = "LANDSCAPE" paper_margin { top = "0.5in" bottom = "0.5in" left = "0.5in" right = "0.5in" } } } } } } content_type = "PAGINATED" } calculated_fields { data_set_identifier = "Bordereau" name = "loss_total" expression = "{loss_payment}+{loss_expense_payment}-{loss_recovery_payment}+{loss_recovery_expense_payment}" } parameter_declarations { date_time_parameter_declaration { name = "PreviousMonth" default_values {

      rolling_date {
        expression = "addDateTime(-1, 'MM', truncDate('MM', now()))"
      }
    }
    time_granularity = "DAY"
  }
}
parameters_declarations {
  date_time_parameter_declaration {
    name = "StartDate"
    default_values {

      rolling_date {
        expression = "addDateTime(-1, 'MM', truncDate('MM', now()))"
      }
    }
    time_granularity = "DAY"
  }
}
parameters_declarations {
  date_time_parameter_declaration {
    name = "EndDate"
    default_values {

      rolling_date {
        expression = "addDateTime(-1, 'SS', truncDate('MM', now()))"
      }
    }
    time_granularity = "DAY"
  }
}
filter_groups {
  filter_group_id = "1fac55b9-d176-4f08-bdcf-300a233c469f"
  filters {
    time_range_filter {
      filter_id = "ac520fc4-2e08-442b-9c00-f349f18f09cc"
      column {
        data_set_identifier = "Bordereau"
        column_name         = "createtime"
      }
      include_minimum = true

      range_minimum_value {
        rolling_date {
          expression = "addDateTime(-1, 'MM', truncDate('MM', now()))"
        }
      }
      range_maximum_value {
        rolling_date {
          expression = "addDateTime(-1, 'SS', truncDate('MM', now()))"
        }
      }
      null_option      = "NON_NULLS_ONLY"
      time_granularity = "DAY"
    }
  }
  scope_configuration {
    selected_sheets {
      sheet_visual_scoping_configurations {
        sheet_id = "a2cfe993-c3ce-4949-b238-5cfbb38eea3e"
        scope    = "ALL_VISUALS"
      }
    }
  }
  status        = "ENABLED"
  cross_dataset = "ALL_DATASETS"
}
column_configurations {
  column {
    data_set_identifier = "Bordereau"
    column_name         = "createtime"
  }
  format_configuration {
    date_time_format_configuration {
      date_time_format = "MM/DD/YYY"
    }
  }
}
analysis_defaults {
  default_new_sheet_configuration {
    interactive_layout_configuration {
      grid {
        canvas_size_options {
          screen_canvas_size_options {
            resize_option             = "FIXED"
            optimized_view_port_width = "1600px"
          }
        }
      }
    }
    paginated_layout_configuration {
      section_based {
        canvas_size_options {
          paper_canvas_size_options {
            paper_size        = "US_LETTER"
            paper_orientation = "LANDSCAPE"
          }
        }
      }
    }
    sheet_content_type = "INTERACTIVE"
  }
}

} }

Steps to Reproduce

Simply run the above TF in any account

Debug Output

No response

Panic Output

No response

Important Factoids

I suggest that you do a rewrite of the resource (and dashboard/analysis) and simply take in the AWS specified JSON as the value for that argument. This will prevent drift between the provider and AWS and will eliminate the complex modeling that I expect is in the provider from a schema perspective.

The AWS definition for templates/dashboards/analysis in QuickSight is exceptionally complex and appears to change frequently. There is almost no way that someone could handcode a template definition other than a trivial example. The simple example that I provided above is ~770 lines long!

The way that we are working with these is to build the analysis using the QuickSight console, then converting It to a template in TF, and then using that template to create dashboards and analyses.

References

No response

Would you like to implement a fix?

No

github-actions[bot] commented 11 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

joseph-wortmann commented 11 months ago

Note that the aws_quicksight_template, aws_quicksight_dashboard and aws_quicksight_analysis are unusable if attempting to use the definition field to create/manage them. To much drift from the current AWS definition of the data structure.

Once again, I would recommend getting rid of the schema for definition and simply allowing the user to pass in a JSON definition, probably extracted from a development version of the analysis/template.

joseph-wortmann commented 11 months ago

Actually, upon further attempts aws_quicksight_template, aws_quicksight_dashboard and aws_quicksight_analysis have proven to be completely unusable due to drift between the schema and the definitions of those objects in AWS.

Even if you create a dashboard from a template, the resulting object will fail as It attempts to parse the dashboard's definition returned by AWS.

As It appears that the definitions for these objects in AWS changes regularly, the only real way to handle this in the provider is to opaquely accept and send the definition as JSON.

Please fix this!

tb102122 commented 11 months ago

@joseph-wortmann you mean something like https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_rest_api with the OpenAPI body?

joseph-wortmann commented 11 months ago

@tb102122 Yes, something like that. The current provider is trying to parse a structure that is even more complex than OpenAPI into a typed schema and It is failing.

Really, the only way to manage the template, analysis and dashboard resources is through extraction of the definition (JSON) manually after developing them in the UI and then using that definition to allow TF to create the resource permanently. The definition can get HUGE.

tb102122 commented 11 months ago

Yes I agree with you I started preparing scripts in python which generate the resources using https://jinja.palletsprojects.com/ but stopped at one stage due to the naming inconsistency and the drift you are referring too. The process for the dataset works quite well so far.

joseph-wortmann commented 11 months ago

aws_quicksight_data_set resources are fully functional and work well.

tb102122 commented 11 months ago

aws_quicksight_data_set resources are fully functional and work well.

Yes it works correctly, I was only referring to the process of generating the resources automatically based on the Console and scripting it afterwards since the QuickSight resources are rather complex.

awseric commented 8 months ago

food for thought: the current implementation looks to be an attempt to generate tf schema from the go sdk inputs. Accepting json as input for definitions would mean marshalling into the go types. I 100% agree that passing definitions as raw json would be easier, and preferable, especially when it comes to reusing existing definition files. However, this does break from the seemingly common standard of mirroring the go sdk's inputs. Currently i have been forced to use a lambda provider fn to manage quicksight resources to get around this. Terraform for quicksight is almost completely useless in its current state.

One of the follow needs to happen IMHO: