search

awslabs / amazon-dynamodb-tools

Tools to make effective use of DynamoDB easier.
Apache License 2.0
113 stars 24 forks source link

Fix reco count days correctly #15

Closed h-minamiohtsu closed 5 months ago

h-minamiohtsu commented 1 year ago

Issue

https://github.com/awslabs/amazon-dynamodb-tools/issues/14

Description of changes:

Fixed an unnecessary warning message by reco due to incorrect counting of days. Prior to the fix, the number of days was calculated from an hour shorter than the actual number of days.

Before fix

$ python3 src/ddbr.py reco --file-name ../APN1.csv --term 1 --output plain
Intepreting CUR data.
Recommendation will be generated between 02/01/23 00:00:00 and 02/28/23 23:00:00 based on source data.
WARNING: The selected start and end times in file have greater or fewer days than a normal month. As a result, the 'monthly' summaries in the output will not reflect the true monthly cost.
Recommendation will be generated for 1 region(s)
Loading CSV into memory. Please wait.
Generating recommendations.
############################################################
DynamoDB Reserved Capacity Report
These recommendations are based on data from 02/01/23 00:00:00 to 02/28/23 23:00:00. The time span represents 27 days of data. RC term length for the report is 1 year(s)
Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode.
Generated on 03/07/23 22:11:02 with reco version v1.1.1
############################################################
************************************************************
WARNING: Non-standard number of days detected in the report. The fields marked monthly in this report represent the total cost for the selected time period, not the true monthly cost.
Re-run the report with data for only one month to receive accurate monthly cost estimates.
************************************************************
************************************************************
               Tokyo region - ap-northeast-1
************************************************************
Capacity type: Reserved Read Capacity Unit (RCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 64,200 RCU(s), which is bought as 642 units of 100. Upfront cost: $21,956.40
    Effective monthly rate: $3,236.88, monthly rate after first month: $1,552.55
    Monthly savings over the public rate card: $2,368.61 (42.26%)
    min: 11,135.0 median: 63,007.0 max: 86,645.0 average: 56,209.56 std_dev: 18,761.16 sum: 37,772,821.0
Capacity type: Reserved Write Capacity Unit (WCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 2,600 WCU(s), which is bought as 26 units of 100. Upfront cost: $4,456.40
    Effective monthly rate: $1,057.84, monthly rate after first month: $715.98
    Monthly savings over the public rate card: $603.55 (36.33%)
    min: 1,662.0 median: 2,519.0 max: 21,694.0 average: 3,331.96 std_dev: 2,849.43 sum: 2,239,074.0
Totals: Effective monthly rate: $4,294.72, monthly rate after first month: $2,268.54, Monthly savings over the public rate card: $2,972.16 (40.9%)
************************************************************
                          Glossary
************************************************************
Reserved capacity to own: How much RC this payer account should own. You must compare the amount to own against your current RC reservations to determine the amount to buy. See report header.
Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect. This amount is only valid if the time range is one month, otherwise this field is actually the effective cost over the time period selected.
Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units. This amount is only valid if the time range is one month, otherwise this field is actually the sum of the cost over the time period selected.
'...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts.
'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.
############################################################
End of report.
Totals: Effective monthly rate: $4,294.72, Total savings over the public rate card for duration of term: $35,665.88 (40.9%), Total upfront purchase cost: $26,412.80
############################################################

After fix

$ python3 src/ddbr.py reco --file-name ../APN1.csv --term 1 --output plain
Intepreting CUR data.
Recommendation will be generated between 02/01/23 00:00:00 and 02/28/23 23:00:00 based on source data.
Recommendation will be generated for 1 region(s)
Loading CSV into memory. Please wait.
Generating recommendations.
############################################################
DynamoDB Reserved Capacity Report
These recommendations are based on data from 02/01/23 00:00:00 to 02/28/23 23:00:00. The time span represents 28 days of data. RC term length for the report is 1 year(s)
Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode.
Generated on 03/07/23 22:09:45 with reco version v1.1.1
############################################################
************************************************************
               Tokyo region - ap-northeast-1
************************************************************
Capacity type: Reserved Read Capacity Unit (RCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 64,200 RCU(s), which is bought as 642 units of 100. Upfront cost: $21,956.40
    Effective monthly rate: $3,236.88, monthly rate after first month: $1,552.55
    Monthly savings over the public rate card: $2,368.61 (42.26%)
    min: 11,135.0 median: 63,007.0 max: 86,645.0 average: 56,209.56 std_dev: 18,761.16 sum: 37,772,821.0
Capacity type: Reserved Write Capacity Unit (WCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 2,600 WCU(s), which is bought as 26 units of 100. Upfront cost: $4,456.40
    Effective monthly rate: $1,057.84, monthly rate after first month: $715.98
    Monthly savings over the public rate card: $603.55 (36.33%)
    min: 1,662.0 median: 2,519.0 max: 21,694.0 average: 3,331.96 std_dev: 2,849.43 sum: 2,239,074.0
Totals: Effective monthly rate: $4,294.72, monthly rate after first month: $2,268.54, Monthly savings over the public rate card: $2,972.16 (40.9%)
************************************************************
                          Glossary
************************************************************
Reserved capacity to own: How much RC this payer account should own. You must compare the amount to own against your current RC reservations to determine the amount to buy. See report header.
Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect. This amount is only valid if the time range is one month, otherwise this field is actually the effective cost over the time period selected.
Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units. This amount is only valid if the time range is one month, otherwise this field is actually the sum of the cost over the time period selected.
'...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts.
'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.
############################################################
End of report.
Totals: Effective monthly rate: $4,294.72, Total savings over the public rate card for duration of term: $35,665.88 (40.9%), Total upfront purchase cost: $26,412.80
############################################################

Diff in output before and after fix

$ diff before_output after_output
3d2
< WARNING: The selected start and end times in file have greater or fewer days than a normal month. As a result, the 'monthly' summaries in the output will not reflect the true monthly cost.
9c8
< These recommendations are based on data from 02/01/23 00:00:00 to 02/28/23 23:00:00. The time span represents 27 days of data. RC term length for the report is 1 year(s)
---
> These recommendations are based on data from 02/01/23 00:00:00 to 02/28/23 23:00:00. The time span represents 28 days of data. RC term length for the report is 1 year(s)
11c10
< Generated on 03/07/23 22:11:02 with reco version v1.1.1
---
> Generated on 03/07/23 22:09:45 with reco version v1.1.1
14,17d12
< WARNING: Non-standard number of days detected in the report. The fields marked monthly in this report represent the total cost for the selected time period, not the true monthly cost.
< Re-run the report with data for only one month to receive accurate monthly cost estimates.
< ************************************************************

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

h-minamiohtsu commented 1 year ago

If the number of days is greater or fewer than the normal month, a warning message will still be displayed 😉 

$ python3 src/ddbr.py reco --file-name APN1_over.csv --term 1 --output plain
Intepreting CUR data.
Recommendation will be generated between 02/01/23 00:00:00 and 03/04/23 23:00:00 based on source data.
WARNING: The selected start and end times in file have greater or fewer days than a normal month. As a result, the 'monthly' summaries in the output will not reflect the true monthly cost.
Recommendation will be generated for 1 region(s)
Loading CSV into memory. Please wait.
Generating recommendations.
############################################################
DynamoDB Reserved Capacity Report
These recommendations are based on data from 02/01/23 00:00:00 to 03/04/23 23:00:00. The time span represents 32 days of data. RC term length for the report is 1 year(s)
Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode.
Generated on 03/07/23 22:30:05 with reco version v1.1.1
############################################################
************************************************************
WARNING: Non-standard number of days detected in the report. The fields marked monthly in this report represent the total cost for the selected time period, not the true monthly cost.
Re-run the report with data for only one month to receive accurate monthly cost estimates.
************************************************************
************************************************************
               Tokyo region - ap-northeast-1
************************************************************
Capacity type: Reserved Read Capacity Unit (RCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 62,200 RCU(s), which is bought as 622 units of 100. Upfront cost: $21,272.40
    Effective monthly rate: $3,650.57, monthly rate after first month: $1,785.59
    Monthly savings over the public rate card: $1,963.26 (34.97%)
    min: 0.0 median: 60,617.5 max: 86,645.0 average: 49,256.6 std_dev: 25,504.22 sum: 37,829,069.0
Capacity type: Reserved Write Capacity Unit (WCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 2,500 WCU(s), which is bought as 25 units of 100. Upfront cost: $4,285.00
    Effective monthly rate: $1,141.01, monthly rate after first month: $765.34
    Monthly savings over the public rate card: $520.38 (31.32%)
    min: 0.0 median: 2,420.5 max: 21,694.0 average: 2,915.46 std_dev: 2,884.2 sum: 2,239,074.0
Totals: Effective monthly rate: $4,791.58, monthly rate after first month: $2,550.93, Monthly savings over the public rate card: $2,483.65 (34.14%)
************************************************************
                          Glossary
************************************************************
Reserved capacity to own: How much RC this payer account should own. You must compare the amount to own against your current RC reservations to determine the amount to buy. See report header.
Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect. This amount is only valid if the time range is one month, otherwise this field is actually the effective cost over the time period selected.
Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units. This amount is only valid if the time range is one month, otherwise this field is actually the sum of the cost over the time period selected.
'...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts.
'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.
############################################################
End of report.
Totals: Effective monthly rate: $4,791.58, Total savings over the public rate card for duration of term: $29,803.72 (34.14%), Total upfront purchase cost: $25,557.40
############################################################
$ python3 src/ddbr.py reco --file-name APN1_less.csv --term 1 --output plain
Intepreting CUR data.
Recommendation will be generated between 02/01/23 00:00:00 and 02/27/23 23:00:00 based on source data.
WARNING: The selected start and end times in file have greater or fewer days than a normal month. As a result, the 'monthly' summaries in the output will not reflect the true monthly cost.
Recommendation will be generated for 1 region(s)
Loading CSV into memory. Please wait.
Generating recommendations.
############################################################
DynamoDB Reserved Capacity Report
These recommendations are based on data from 02/01/23 00:00:00 to 02/27/23 23:00:00. The time span represents 27 days of data. RC term length for the report is 1 year(s)
Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode.
Generated on 03/07/23 22:29:54 with reco version v1.1.1
############################################################
************************************************************
WARNING: Non-standard number of days detected in the report. The fields marked monthly in this report represent the total cost for the selected time period, not the true monthly cost.
Re-run the report with data for only one month to receive accurate monthly cost estimates.
************************************************************
************************************************************
               Tokyo region - ap-northeast-1
************************************************************
Capacity type: Reserved Read Capacity Unit (RCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 64,200 RCU(s), which is bought as 642 units of 100. Upfront cost: $21,956.40
    Effective monthly rate: $3,124.59, monthly rate after first month: $1,500.42
    Monthly savings over the public rate card: $2,282.32 (42.21%)
    min: 11,135.0 median: 63,103.5 max: 86,645.0 average: 56,226.44 std_dev: 18,810.75 sum: 36,434,734.0
Capacity type: Reserved Write Capacity Unit (WCU), Term: 1 year(s)
    Reserved capacity to own (not necessarily buy): 2,600 WCU(s), which is bought as 26 units of 100. Upfront cost: $4,456.40
    Effective monthly rate: $1,029.73, monthly rate after first month: $700.08
    Monthly savings over the public rate card: $583.35 (36.16%)
    min: 1,662.0 median: 2,531.5 max: 21,694.0 average: 3,354.88 std_dev: 2,888.94 sum: 2,173,964.0
Totals: Effective monthly rate: $4,154.32, monthly rate after first month: $2,200.50, Monthly savings over the public rate card: $2,865.68 (40.82%)
************************************************************
                          Glossary
************************************************************
Reserved capacity to own: How much RC this payer account should own. You must compare the amount to own against your current RC reservations to determine the amount to buy. See report header.
Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect. This amount is only valid if the time range is one month, otherwise this field is actually the effective cost over the time period selected.
Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units. This amount is only valid if the time range is one month, otherwise this field is actually the sum of the cost over the time period selected.
'...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts.
'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.
############################################################
End of report.
Totals: Effective monthly rate: $4,154.32, Total savings over the public rate card for duration of term: $34,388.11 (40.82%), Total upfront purchase cost: $26,412.80
############################################################
switch180 commented 5 months ago

Apologies - I completely missed this PR. LGTM. Merging.