Question regarding reserved instances and ice wildly over-reporting costs

[rayrod2030]

I'm seeing a situation in the last two to three months where the costs that ice is reporting are drastically higher than the billing summaries we are seeing within the AWS console. This happens to coincide with a large purchase of reserved resources (rds, ec2, elasticache) that we made around that same time that our reporting starting looking way off in ice. Currently we are seeing reported costs in the ballpark of 28% over actual aws reported costs in ice for the last few months.

Is it safe to assume that if we have not properly configured ice to account for our reserved instances that we'll see incorrect (as in much higher numbers) cost rollups within ice?

Also I've verified these numbers are way over for us both with standard billing files and our detailed billing files with tags and resources.

[fangji]

@rayrod2030 Can you help me find out the cause of this? First, can you compare rds, ec2 and elasticache cost between Ice and your standard billing files and let me know the difference? If you see a cost difference can you also find out the usage difference?

Thanks -Fang

[rayrod2030]

@fangji These are the numbers I've analyzed for the last 3 months for EC2 (ec2, ec2_instance, ebs, eip, cloudwatch), RDS and Elasticache in ice and our aws billing as reported in our primary billing account. As you can see we made some reserved purchases in august and that seems to have effected EC2 ice costs the most. Appreciate your help with this.

[fangji]

@rayrod2030 Thanks for providing the data.

For the ec2 cost, I assume it includes both ec2 and ec2_instance? Is the difference mainly in ec2_instance?

Does the ec2 aws billing include amortized upfront cost?

If the difference is mainly in ec2_instance, can you confirm the ec2_instance usage hours still match between Ice and the billing?

Thanks -Fang

[rayrod2030]

@fangji For my rollups in my spreadsheet I combined ec2_instance, ec2, ebs, cloudwatch and eip ice resources as those are the resources that AWS combines under the "Amazon Elastic Compute Cloud" line item. I used those combined totals for comparison.

Our AWS billing summaries in our main payment account does show upfront costs for reserved purchases for August and September as well as showing 0.00 cost line items for our reserved instance usage for the last two months. I suspect if I were to calculate the usage using the standard on-demand pricing that it should make up for the large difference in ec2_instance costs in ice.

I can confirm that since august our ec2 costs within ice have been pretty stable and all increases we have seen in ice are in the ec2_instance resource.

If anything our ec2_instance usage is slightly higher in our aws billing summary but not by that much.

So am I basically seeing a discrepancy due to the fact that ice is processing my reserved instance usage at the on-demand price?

[fangji]

@rayrod2030 Do you have ice.reservationCapacityPoller=true in ice.properties file? In reservations page on UI, can you see if the number of reserved instances match with what you have?

[rayrod2030]

@fangji Currently we are not running reservationCapacityPoller in production as I was not sure if it was needed to report proper reservation pricing or if it enabling it would just give you move visibility into how many of your reserve instance capacity was being used. I'm assuming now that if we don't have this enabled we won't be seeing proper costs based on reserved instance pricing?

[fangji]

@rayrod2030 If you are not running reservationCapacityPoller, then Ice should show reserved&ondemand instance usage&cost as it is in billing file.

Did you define reservation owner accounts? Do you have properties like ice.owneraccount.* in your ice.properties file and what values do those properties have?

[rayrod2030]

@fangji I believe I do have reservation owner accounts defined. Here is what my ice properties file looks like for our processor. I've edited it to scrub any real account information.

#
## Ice Configuration File
#
## Dynamically generated by Chef
##
## Local modifications will be overwritten by Chef.
##

ice.processor=true
ice.processor.localDir=/var/ice_processor

ice.billing_s3bucketname=billing-stuff
ice.billing_s3bucketprefix=

ice.reader=false
ice.reader.localDir=/var/ice_reader

ice.work_s3bucketname=ice-work-files
ice.work_s3bucketprefix=

ice.startmillis=1367294400000

ice.account.green=111111111111
ice.account.blue=222222222222
ice.account.pink=333333333333
ice.account.red=444444444444
ice.account.yellow=555555555555
ice.account.gray=666666666666
ice.account.purple=777777777777
ice.account.black=888888888888

ice.owneraccount.black=green,blue,pink,red,yellow,gray,purple

ice.reservationCapacityPoller=false

# # reservation period, possible values are oneyear, threeyear
ice.reservationPeriod=threeyear
# # reservation utilization, possible values are LIGHT, HEAVY
ice.reservationUtilization=HEAVY

ice.companyName=Fizzbuzz

ice.customTags=user:Product,user:Environment,user:Type

[fangji]

@rayrod2030 If you don't have reservation capacity poller enabled, please remove ice.owneraccount.* properties. Try that and re-process the data.