turbot / steampipe-plugin-aws

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/aws
Apache License 2.0
177 stars 90 forks source link

Updated the aws_organizations_account table design to resolve the cache issue and to resolve the query based on query plan Closes #2235 #2236

Closed ParthaI closed 16 hours ago

ParthaI commented 2 days ago
Add passing integration test logs here

Example query results

Results Case 1: CTE and the select query is querying for a single connection ``` > with cost_data as ( select linked_account_id, period_start, unblended_cost_amount as this_month, lag(unblended_cost_amount, 1) over( partition by linked_account_id order by period_start desc ) as previous_month from aws_cost_by_account_monthly ) select linked_account_id, period_start, this_month :: numeric :: float, previous_month :: numeric :: float, ((this_month + previous_month) / 2) :: numeric :: money as "60_day_avg" from cost_data join aws_organizations_account as acc on acc.id = cost_data.linked_account_id where acc.parent_id = 'r-bhwu' order by linked_account_id, period_start; +-------------------+---------------------------+----------------+----------------+------------+ | linked_account_id | period_start | this_month | previous_month | 60_day_avg | +-------------------+---------------------------+----------------+----------------+------------+ | xxxxxxxxxxxx | 2023-07-03T05:30:00+05:30 | 349.2449938927 | 33.2081890726 | $191.23 | | xxxxxxxxxxxx | 2023-08-01T05:30:00+05:30 | 33.2081890726 | 123.6535233683 | $78.43 | | xxxxxxxxxxxx | 2023-09-01T05:30:00+05:30 | 123.6535233683 | -0.0000147239 | $61.83 | | xxxxxxxxxxxx | 2023-10-01T05:30:00+05:30 | -0.0000147239 | 150.1021639716 | $75.05 | | xxxxxxxxxxxx | 2023-11-01T05:30:00+05:30 | 150.1021639716 | 148.9409804602 | $149.52 | | xxxxxxxxxxxx | 2023-12-01T05:30:00+05:30 | 148.9409804602 | 144.4520678504 | $146.70 | | xxxxxxxxxxxx | 2024-01-01T05:30:00+05:30 | 144.4520678504 | 143.5126080015 | $143.98 | | xxxxxxxxxxxx | 2024-02-01T05:30:00+05:30 | 143.5126080015 | 214.8022529982 | $179.16 | | xxxxxxxxxxxx | 2024-03-01T05:30:00+05:30 | 214.8022529982 | 146.9339251547 | $180.87 | | xxxxxxxxxxxx | 2024-04-01T05:30:00+05:30 | 146.9339251547 | 110.3291507393 | $128.63 | | xxxxxxxxxxxx | 2024-05-01T05:30:00+05:30 | 110.3291507393 | 98.49682533 | $104.41 | | xxxxxxxxxxxx | 2024-06-01T05:30:00+05:30 | 98.49682533 | 5.0175839387 | $51.76 | | xxxxxxxxxxxx | 2024-07-01T05:30:00+05:30 | 5.0175839387 | | | +-------------------+---------------------------+----------------+----------------+------------+ Time: 2.5s. Rows returned: 13. Rows fetched: 26 (13 cached). Hydrate calls: 0. Scans: 14. Scans: 1) aws_cost_by_account_monthly.aws: Time: 1.7s. Fetched: 13. Hydrates: 0. 2) aws_organizations_account.aws: Time: 6ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 3) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 4) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 5) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 6) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 7) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 8) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 9) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 10) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 11) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 12) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 13) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 14) aws_organizations_account.aws: Time: 3ms. Fetched: 1 (cached). Hydrates: 0. Quals: id=xxxxxxxxxxxx, parent_id=r-bhwu, parent_id=r-bhwu. ``` Case 2: CTE(Member account connection) and the select query is with (Organization account connection) ``` with cost_data as ( select linked_account_id, period_start, unblended_cost_amount as this_month, lag(unblended_cost_amount, 1) over( partition by linked_account_id order by period_start desc ) as previous_month from aws_aab.aws_cost_by_account_monthly ) select linked_account_id, period_start, this_month :: numeric :: float, previous_month :: numeric :: float, ((this_month + previous_month) / 2) :: numeric :: money as "60_day_avg" from cost_data join aws_organizations_account as acc on acc.id = cost_data.linked_account_id where acc.parent_id = 'r-bhwu' order by linked_account_id, period_start; +-------------------+---------------------------+----------------+----------------+------------+ | linked_account_id | period_start | this_month | previous_month | 60_day_avg | +-------------------+---------------------------+----------------+----------------+------------+ | xxxxxxxxxxxx | 2023-07-03T05:30:00+05:30 | 183.4736317556 | 167.8078015321 | $175.64 | | xxxxxxxxxxxx | 2023-08-01T05:30:00+05:30 | 167.8078015321 | 52.0589675091 | $109.93 | | xxxxxxxxxxxx | 2023-09-01T05:30:00+05:30 | 52.0589675091 | -0.0000466359 | $26.03 | | xxxxxxxxxxxx | 2023-10-01T05:30:00+05:30 | -0.0000466359 | 79.2983828174 | $39.65 | | xxxxxxxxxxxx | 2023-11-01T05:30:00+05:30 | 79.2983828174 | 92.2740219081 | $85.79 | | xxxxxxxxxxxx | 2023-12-01T05:30:00+05:30 | 92.2740219081 | 99.9622985376 | $96.12 | | xxxxxxxxxxxx | 2024-01-01T05:30:00+05:30 | 99.9622985376 | 51.7028170697 | $75.83 | | xxxxxxxxxxxx | 2024-02-01T05:30:00+05:30 | 51.7028170697 | 99.7647663327 | $75.73 | | xxxxxxxxxxxx | 2024-03-01T05:30:00+05:30 | 99.7647663327 | 104.3970835384 | $102.08 | | xxxxxxxxxxxx | 2024-04-01T05:30:00+05:30 | 104.3970835384 | 66.8429768983 | $85.62 | | xxxxxxxxxxxx | 2024-05-01T05:30:00+05:30 | 66.8429768983 | 72.5401955491 | $69.69 | | xxxxxxxxxxxx | 2024-06-01T05:30:00+05:30 | 72.5401955491 | 4.6981874099 | $38.62 | | xxxxxxxxxxxx | 2024-07-01T05:30:00+05:30 | 4.6981874099 | | | +-------------------+---------------------------+----------------+----------------+------------+ Time: 9.8s. Rows returned: 13. Rows fetched: 26. Hydrate calls: 0. Scans: 14. Connections: 2. Scans: 1) aws_cost_by_account_monthly.aws_aab: Time: 1.6s. Fetched: 13. Hydrates: 0. 2) aws_organizations_account.aws: Time: 1.3s. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 3) aws_organizations_account.aws: Time: 1.1s. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 4) aws_organizations_account.aws: Time: 494ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 5) aws_organizations_account.aws: Time: 475ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 6) aws_organizations_account.aws: Time: 470ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 7) aws_organizations_account.aws: Time: 467ms. Fetched: 1. Hydrates: 0. Quals: id=xxxxxxxxxxxx, parent_id=r-bhwu, parent_id=r-bhwu. 8) aws_organizations_account.aws: Time: 467ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 9) aws_organizations_account.aws: Time: 464ms. Fetched: 1. Hydrates: 0. Quals: id=xxxxxxxxxxxx, parent_id=r-bhwu, parent_id=r-bhwu. 10) aws_organizations_account.aws: Time: 454ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 11) aws_organizations_account.aws: Time: 447ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 12) aws_organizations_account.aws: Time: 423ms. Fetched: 1. Hydrates: 0. Quals: id=xxxxxxxxxxxx, parent_id=r-bhwu, parent_id=r-bhwu. 13) aws_organizations_account.aws: Time: 420ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. 14) aws_organizations_account.aws: Time: 415ms. Fetched: 1. Hydrates: 0. Quals: parent_id=r-bhwu, parent_id=r-bhwu, id=xxxxxxxxxxxx. ``` ``` > select id,parent_id,title from aws_organizations_account WHERE parent_id IN (select id from aws_organizations_organizational_unit WHERE parent_id='ou-wxnb-wofu2g1q') LIMIT 2 +--------------+------------------+--------------+ | id | parent_id | title | +--------------+------------------+--------------+ | 060123323170 | ou-xqk-9o38j3vkp | aaa @ zaphod | | 934055246342 | ou-xqk-9o38j3vkp | aac @ zaphod | +--------------+------------------+--------------+ ``` ``` > select id, parent_id, title from aws_organizations_account where parent_id in ( select id from aws_organizations_organizational_unit where parent_id = 'ou-wxnb-wofu2g1q' ) order by id, parent_id, title limit 2; +--------------+------------------+--------------+ | id | parent_id | title | +--------------+------------------+--------------+ | 060123323170 | ou-xqk-9o38j3vkp | aaa @ zaphod | | 934055246342 | ou-xqk-9o38j3vkp | aac @ zaphod | +--------------+------------------+--------------+ ```