cid-cmd deploy: Athena query failed: Column 'product_instance_family' cannot be resolved

[ivan-kovnatskyi-c54f97a]

Hello, thanks for working on these dashboards and infrastructure to automate configuring all this.

I've deployed underlying resources using this code:

module "cur_source" {
  source = "github.com/aws-samples/aws-cudos-framework-deployment//terraform-modules/cur-setup-source?ref=0.3.9"

  destination_bucket_arn = module.cur_destination.cur_bucket_arn

  enable_split_cost_allocation_data = true

  # Provider alias for us-east-1 must be passed explicitly (required for CUR setup)
  providers = {
    aws.useast1 = aws.useast1
  }
}

module "cur_destination" {
  source = "github.com/aws-samples/aws-cudos-framework-deployment//terraform-modules/cur-setup-destination?ref=0.3.9"

  source_account_ids = ["${local.aws_account_id}"]
  create_cur         = false # Set to true to create an additional CUR in the aggregation account

  # Provider alias for us-east-1 must be passed explicitly (required for CUR setup)
  providers = {
    aws.useast1 = aws.useast1
  }
}

module "cid_dashboards" {
  source = "github.com/aws-samples/aws-cudos-framework-deployment//terraform-modules/cid-dashboards?ref=0.3.9"

  stack_name      = "Cloud-Intelligence-Dashboards"
  template_bucket = local.s3_bucket

  stack_parameters = {
    "PrerequisitesQuickSight"            = "yes"
    "PrerequisitesQuickSightPermissions" = "yes"
    "QuickSightUser"                     = "AWSReservedSSO_[redacted]/[redacted].com"

    # Somehow when you remove configuring dashboards you won't have Glue
    # Crawler, Athena DB and other resources created, so I need to deploy
    # dashboards to be able to use cid-cmd tool to deploy SCAD dashboard.
    "DeployCUDOSDashboard"               = "yes"
    "DeployCostIntelligenceDashboard"    = "yes"
    "DeployKPIDashboard"                 = "yes"
  }
}

While waiting for data export to be ready in 24 hours, decided to deploy SCAD dashboard and getting this error:

./.venv/bin/cid-cmd deploy --dashboard-id scad-containers-cost-allocation
CLOUD INTELLIGENCE DASHBOARDS (CID) CLI 0.3.9 Beta

Loading plugins...
    Core loaded

Checking AWS environment...
    profile name: [redacted]
    accountId: [redacted]
    AWS userId: AWSReservedSSO_[redacted]/[redacted]@[redacted].com
    Region: us-east-1

Latest template: arn:aws:quicksight:us-east-1:223485597511:template/scad-containers-cost-allocation/version/2
Dashboard "scad-containers-cost-allocation" is not deployed

Required datasets: 
 - scad_cca_summary_view
 - scad_cca_hourly_resource_view

Looking by DataSetId defined in template...complete

There are still 2 datasets missing: scad_cca_hourly_resource_view, scad_cca_summary_view
Creating dataset: scad_cca_hourly_resource_view

? [athena-workgroup] Select Amazon Athena workgroup to use: CID

? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): CID-Athena CID-Athena (workgroup=CID)

? [athena-database] Select AWS Athena database to use: cid_cur
Detected views: account_map
Missing views: scad_cca_hourly_resource_view
Checking if CUR is enabled and available...

? [cur-table-name] Please select CUR: cur
    Athena table: cur
    Resource IDs: yes
    SavingsPlans: yes
    Reserved Instances: yes

^[[AColumn 'split_line_item_parent_resource_id' was added to CUR (cur).
Column 'split_line_item_reserved_usage' was added to CUR (cur).
Column 'split_line_item_actual_usage' was added to CUR (cur).
Column 'split_line_item_split_usage' was added to CUR (cur).
Column 'split_line_item_split_usage_ratio' was added to CUR (cur).
Column 'split_line_item_split_cost' was added to CUR (cur).
Column 'split_line_item_unused_cost' was added to CUR (cur).
Column 'split_line_item_net_split_cost' was added to CUR (cur).
Column 'split_line_item_net_unused_cost' was added to CUR (cur).
Column 'split_line_item_public_on_demand_split_cost' was added to CUR (cur).
Column 'split_line_item_public_on_demand_unused_cost' was added to CUR (cur).
Column 'resource_tags_aws_eks_cluster_name' was added to CUR (cur).
Column 'resource_tags_aws_eks_deployment' was added to CUR (cur).
Column 'resource_tags_aws_eks_namespace' was added to CUR (cur).
Column 'resource_tags_aws_eks_node' was added to CUR (cur).
Column 'resource_tags_aws_eks_workload_name' was added to CUR (cur).
Column 'resource_tags_aws_eks_workload_type' was added to CUR (cur).
CRITICAL - Athena query failed: line 77:16: Column 'product_instance_family' cannot be resolved

In the table I see this column: product_instance_type_family

Do I miss anything maybe?

Would be helpful for any advices how to mitigate that.

[ivan-kovnatskyi-c54f97a]

#######
####### Congratulations!
####### SCAD Containers Cost Allocation is available at: https://us-east-1.quicksight.aws.amazon.com/sn/dashboards/scad-containers-cost-allocation
#######

? [share-with-account] Share this dashboard with everyone in the account?: no

I was able to create later on. Probably data populated and it got needed columns.

[ivan-kovnatskyi-c54f97a]

Sorry for re-opening, but for the reason not to loose context, regarding this one:

    # Somehow when you remove configuring dashboards you won't have Glue
    # Crawler, Athena DB and other resources created, so I need to deploy
    # dashboards to be able to use cid-cmd tool to deploy SCAD dashboard.
    "DeployCUDOSDashboard"               = "yes"
    "DeployCostIntelligenceDashboard"    = "yes"
    "DeployKPIDashboard"                 = "yes"

Can you please comment on it?

Do I also understand correctly that SCAD dashboard does not depend on the dashboards above? If so, is there any more infra as code solution to deploy scad in scope of terraform, avoiding using cid-cmd tool? As we probably would omit using it in our infra CI.

Thanks.

[iakov-aws]

Yes, the crawler and table is conditional to existance of foundational by design

[ivan-kovnatskyi-c54f97a]

Thanks, I see.