search

aws-samples / aws-cudos-framework-deployment

Command Line Interface tool for Cloud Intelligence Dashboards deployment
https://catalog.workshops.aws/awscid
MIT No Attribution
406 stars 162 forks source link

Unable to deploy AWS budget dashboard - Column 'collection_time' cannot be resolved #986

Open work-bw opened 3 days ago

work-bw commented 3 days ago

Trying to deploy the Budgets dashboard via cid-cmd deploy. I need some help resolving the installation issue please. Thanks.

Brian

=======================

? [dashboard-id] Please select a dashboard to deploy:   [aws-budgets] AWS Budgets
Dashboard "aws-budgets" is not deployed

Required datasets: 
 - budgets_view

Looking by DataSetId defined in template...complete

There are still 1 datasets missing: budgets_view
Creating dataset: budgets_view

? [athena-workgroup] Select Amazon Athena workgroup to use: CID

? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): CID-CMD-Athena <CREATE NEW DATASOURCE>

? [quicksight-datasource-role] Please choose a QuickSight role. It must have access to Athena: CidCmdQuickSightDataSourceRole

? [athena-database] Select AWS Athena database to use: athenacurcfn_nelnet_cudos
Role CidCmdQuickSightDataSourceRole was updated. https://console.aws.amazon.com/iam/home?#/roles/details/CidCmdQuickSightDataSourceRole
ERROR - Data source already exists {'Arn': 'arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena', 'DataSourceId': 'CID-CMD-Athena', 'Name': 'CID-CMD-Athena', 'Type': 'ATHENA', 'Status': 'CREATION_SUCCESSFUL', 'CreatedTime': datetime.datetime(2023, 7, 3, 19, 3, 21, 937000, tzinfo=tzlocal()), 'LastUpdatedTime': datetime.datetime(2023, 7, 3, 19, 3, 23, 202000, tzinfo=tzlocal()), 'DataSourceParameters': {'AthenaParameters': {'WorkGroup': 'primary'}}, 'SslProperties': {'DisableSsl': False}}
athena_datasource <cid.helpers.quicksight.datasource.Datasource object at 0x7fafd68835e0>
Detected views: account_map
Missing views: budgets_view
CRITICAL - Athena query failed: line 12:3: Column 'collection_time' cannot be resolved

====================

When I'm prompted to select the quicksight-datasource-id my only choice is to create new datasource. It doesn't seem to see the existing datasources.

? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): (Use arrow keys)
 » CID-CMD-Athena <CREATE NEW DATASOURCE>
iakov-aws commented 3 days ago

Hello,

Collection time must be there in table : https://github.com/awslabs/cid-framework/blob/main/data-collection/deploy/module-budgets.yaml#L165

Can you check the version of data collection (in description of the stack) and update to the latest https://catalog.workshops.aws/awscid/en-US/data-collection/update

iakov-aws commented 3 days ago

Brian @work-bw Regarding datasources, can you check if datasource status are ok?

some what like this in cloud shell:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
aws quicksight list-data-sources  --aws-account-id $AWS_ACCOUNT_ID --query 'DataSources[?Type==`ATHENA`]'
work-bw commented 2 days ago

I updated the data collection stack to v3.4.3 as I was on an old stack previously. collection_time is not in the table, is that something I need to manually add? If so, how? Can you send me some instructions please?

I think my data sources are okay but I still get the same error trying to install the Budgets dashboard after updating the data collection stack. However, if I need to wait a day for collection_time to be added, it has not been a day yet.

Brian

[cloudshell-user@ip-10-142-89-131 ~]$ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
[cloudshell-user@ip-10-142-89-131 ~]$ aws quicksight list-data-sources  --aws-account-id $AWS_ACCOUNT_ID --query 'DataSources[?Type==`ATHENA`]'
[
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/12693f2e-cd40-4123-ab1d-a0bfadc17ea3",
        "DataSourceId": "12693f2e-cd40-4123-ab1d-a0bfadc17ea3",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-09-25T18:42:08.644000+00:00",
        "LastUpdatedTime": "2019-09-25T18:42:08.644000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/3654057d-3d37-4188-ba9c-82815371ee8e",
        "DataSourceId": "3654057d-3d37-4188-ba9c-82815371ee8e",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-08-15T18:55:42.047000+00:00",
        "LastUpdatedTime": "2019-08-15T18:55:42.047000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/7b112161-8cc9-451a-8b9e-37130c693252",
        "DataSourceId": "7b112161-8cc9-451a-8b9e-37130c693252",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2020-02-06T20:49:19.556000+00:00",
        "LastUpdatedTime": "2020-02-06T20:49:19.556000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/a9344e2a-f64a-4784-8965-558886369da9",
        "DataSourceId": "a9344e2a-f64a-4784-8965-558886369da9",
        "Name": "athenacurcfn_nelnet_e_d_p",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2019-09-25T18:43:42.738000+00:00",
        "LastUpdatedTime": "2019-09-25T18:43:42.738000+00:00",
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/4c587d46-19a8-4cf7-91c6-2f25a55207d1",
        "DataSourceId": "4c587d46-19a8-4cf7-91c6-2f25a55207d1",
        "Name": "cod2",
        "Type": "ATHENA",
        "Status": "UPDATE_SUCCESSFUL",
        "CreatedTime": "2023-07-03T18:33:23.121000+00:00",
        "LastUpdatedTime": "2023-07-03T18:48:37.334000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    },
    {
        "Arn": "arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena",
        "DataSourceId": "CID-CMD-Athena",
        "Name": "CID-CMD-Athena",
        "Type": "ATHENA",
        "Status": "CREATION_SUCCESSFUL",
        "CreatedTime": "2023-07-03T19:03:21.937000+00:00",
        "LastUpdatedTime": "2023-07-03T19:03:23.202000+00:00",
        "DataSourceParameters": {
            "AthenaParameters": {
                "WorkGroup": "primary"
            }
        },
        "SslProperties": {
            "DisableSsl": false
        }
    }
]
work-bw commented 2 days ago

I should also add that I'm getting this same error when I tried to reinstall the trusted advisor organizational dashboard. I deleted the dashboard and when I tried to re-deploy it I get the same error. It always prompts to create a new CID-CMD-Athena datasource and it already exists and throws an error. This looks like a bug reading existing datasources. Thanks.

Brian

? [dashboard-id] Please select a dashboard to deploy:   [ta-organizational-view] Trusted Advisor Organizational View

Latest template: arn:aws:quicksight:us-east-1:223485597511:template/ta-organizational-view/version/19
Dashboard "ta-organizational-view" is not deployed

Required datasets: 
 - ta-organizational-view

Looking by DataSetId defined in template...complete

There are still 1 datasets missing: ta-organizational-view
Creating dataset: ta-organizational-view

? [athena-workgroup] Select Amazon Athena workgroup to use: CID

`? [quicksight-datasource-id] Please choose DataSource (Select the first one if not sure): CID-CMD-Athena <CREATE NEW DATASOURCE>`

? [quicksight-datasource-role] Please choose a QuickSight role. It must have access to Athena: CidCmdQuickSightDataSourceRole

? [athena-database] Select AWS Athena database to use: athenacurcfn_nelnet_cudos
Role CidCmdQuickSightDataSourceRole was updated. https://console.aws.amazon.com/iam/home?#/roles/details/CidCmdQuickSightDataSourceRole
ERROR - Data source already exists {'Arn': 'arn:aws:quicksight:us-west-2:704691252361:datasource/CID-CMD-Athena', 'DataSourceId': 'CID-CMD-Athena', 'Name': 'CID-CMD-Athena', 'Type': 'ATHENA', 'Status': 'CREATION_SUCCESSFUL', 'CreatedTime': datetime.datetime(2023, 7, 3, 19, 3, 21, 937000, tzinfo=tzlocal()), 'LastUpdatedTime': datetime.datetime(2023, 7, 3, 19, 3, 23, 202000, tzinfo=tzlocal()), 'DataSourceParameters': {'AthenaParameters': {'WorkGroup': 'primary'}}, 'SslProperties': {'DisableSsl': False}}
athena_datasource <cid.helpers.quicksight.datasource.Datasource object at 0x7fcdc1d1c610>
Detected views: 
Missing views: ta_org_view
Missing dependency view: ta_organizational_view_reports, creating
{'account_id': '704691252361'}

? [view-ta-organizational-view-reports-s3FolderPath] Required parameter: s3FolderPath (Trusted Advisor Organization View report S3 path): s3://cid-data-704691252361/tr
usted-advisor/trusted-advisor-data

? [timezone] Please select timezone for datasets scheduled refresh.: America/Chicago
Dataset "ta-organizational-view" created
Deploying dashboard ta-organizational-view

#######
####### Congratulations!
####### Trusted Advisor Organizational View is available at: https://us-west-2.quicksight.aws.amazon.com/sn/dashboards/ta-organizational-view
#######