Metric `node_total_hourly_cost` provides underestimated value compared to Azure effective unit price

[jerwen]

For reproducible bugs, please provide the following

Describe the bug

After configuring the Rate Card API and Azure cloud integration, I noticed that the metric node_total_hourly_cost was giving underestimated value, compared to effective unit Price found on my Azure bill.

To Reproduce

Steps to reproduce the behavior:

1. Configure Rate Card API
2. Configure Azure Cloud integration
3. Inspect node_total_hourly_cost from the prometheus instance configured

Expected behavior

With the Rate Card API configured, we would expect the value from the metric to be accurate

Screenshots

Here is table with some discrepancies I noticed, featuring the value from the metric and the unit Price extracted from my Azure bill	Instance Type	node_total_hourly_cost	Azure effective unit price ($)
Standard_D2s_v4	0.09444871109771728	0.112
Standard_D4_v4	0.18890274391937256	0.224
Standard_D4s_v4	0.18890274391937256	0.224
Standard_E4_v4	0.24959945144653323	0.296
Standard_E4_v5	0.24959945144653323	0.296
Standard_E2_v4	0.12481382036972047	0.148
Standard_E2_v3	0.015607435409545898	0.156
Standard_E4_v3	0.03128547882843018	0.031
Standard_E16s_v4	0.9984294322814942	1.184

You can notice that Standard_E4_v3 has an accurate value. This is the only accurate value and it is the only instance that was a spot instance.

┆Issue is synchronized with this Jira Task by Unito

[dwbrown2]

@jerwen do you have an EDP, reserved instances, or anything else applied to these? Also, what region are you in?

[dwbrown2]

Also, where are you getting effective pricing because I'm not seeing list prices match. Here's one example:

[jerwen]

Hello @dwbrown2 , sorry for the delay, I didn't receive notifications on your reply.

@jerwen do you have an EDP, reserved instances, or anything else applied to these? Also, what region are you in?

We do not have reserved instances. We are on Pas as you go plan and the region is "France Central".

And I can confirm that the pricing match at least for the D2s v4 instance type

[jerwen]

@dwbrown2 After doing some comparisons between the values I'm receiving from the metric and the pricing for the North Central US region, I found that they're actually pretty close. This finding leads me to think that there, maybe, is a misconfiguration on the region for the part that's generating the metric value.

[kc-adawson]

I have deployed an Azure cluster in France Central and I see the following in the Kubecost azurenodepricing.json which is similar to the original post. in this thread: "d2sv4": 0.096,

On my test France Central cluster, I'm running Standard Ds2v2 nodes, which I see in the Kubecost azurenodeprices.json as "ds2v2": 0.114, but I see it on the Azure pricing calculator as $0.175/hour.

I also get many warning logs from cost-model around the following:

2022-11-04T16:53:59.526122782Z WRN no pricing data found for francecentral,Standard_DS2_v2,ondemand

No rate card has been configured on my testing cluster since I am trying to match the on-demand rates.

[kc-adawson]

Further Interesting data - the pricing seems to match East US 2 and West US 2 Pricing, as shown below, even though the region is France Central

[kc-adawson]

Confirmed many of the rate card entires line up with East US 2 and not France Central - although there are some instance types in the A range that don't match, so its not 100%. Whats the best path to get these rates corrected?

[dwbrown2]

Sounds like a potential bug, @AjayTripathy @Adam-Stack-PM could you please help triage?

[kc-adawson]

We had another user contact us about Azure pricing being off - This user had a cluster in Southeast Asia, with rate card configured. I had the user check the /var/config/nodeprices.json, and it was the same as the one we are getting for Central France (rates close to East US 2). Seems like some regions are defaulting to an incorrect set of on-demand rates.

[AjayTripathy]

Thanks for the repros @kc-adawson -- we'll get this fixed for v1.99

[srpomeroy]

Engineering can use the following command to add the cluster to their local kubectl config from the Azure sandbox subscription: az aks get-credentials -n adawson-labs-tester -g adawson-labs

[kaelanspatel]

Hi all, did some digging into this issue and found a couple things.

First is that the rate reported in the metrics when azureBillingRegion is set to "FR" and currencyCode is set to "EUR" is consistent with the returned Rate Card API response:

{
    "EffectiveDate": "2020-05-01T00:00:00Z",
    "IncludedQuantity": 0.0,
    "MeterCategory": "Virtual Machines",
    "MeterId": "ddbc7f09-0598-53f6-9ba3-ef8f53c892f1",
    "MeterName": "D2s v4",
    "MeterRates": {
        "0": 0.094449
    },
    "MeterRegion": "FR Central",
    "MeterSubCategory": "Dsv4 Series",
    "MeterTags": [],
    "Unit": "1 Hour"
},

I think this was how @kc-adawson had it set up when doing the above testing, at least.

Whether this is a bug, or is just an incorrect/outdated conversion (0.094 Euros to 0.112 USD is a plausible exchange rate for mid-2020) I'm not sure. But it does appear that the issue is that the raw rate card data is wrong, not some sort of internal parsing error.

Changing the configs/query opts to be azureBillingRegion "US" and currencyCode "USA" gives the expected cost in USD:

{
    "EffectiveDate": "2020-05-01T00:00:00Z",
    "IncludedQuantity": 0.0,
    "MeterCategory": "Virtual Machines",
    "MeterId": "ddbc7f09-0598-53f6-9ba3-ef8f53c892f1",
    "MeterName": "D2s v4",
    "MeterRates": {
        "0": 0.112
    },
    "MeterRegion": "FR Central",
    "MeterSubCategory": "Dsv4 Series",
    "MeterTags": [],
    "Unit": "1 Hour"
},

Depending on if this is an error in the rate card API or merely an outdated price in Euros, this could be WAI or could point to using a non-USD currency in the API opts being inadvisable.

[srpomeroy]

@kaelanspatel sounds like we(or the customer) should open a ticket with Microsoft support. Do you agree? Or is there another action you would suggest?

[kaelanspatel]

I think if we can open a ticket just to get some clarity, that would be great.

[AjayTripathy]

To add some more context, we're pretty sure that 0.9449 is the right number for users actually buying with EUR and region FR. I guess if you're buying with USD and purchasing from region US, it might be 0.112 but I am less sure on this point. @alexkubecost is there someone on the microsoft team who could answer this?

[kaelanspatel]

I think it does depend on conversion rates (or something else), however, as note that when you switch the online calculator to EUR you get 0.108.

[srpomeroy]

That online calculator is likely using the new pricing api.

On Fri, Dec 2, 2022 at 2:21 PM Kaelan Patel @.***> wrote:

I think it does depend on conversion rates (or something else), however, as note that when you switch the online calculator to EUR you get 0.108. [image: image] https://user-images.githubusercontent.com/32113845/205369444-6c35c7fa-a90f-4461-921d-2c628ef40e8c.png

— Reply to this email directly, view it on GitHub https://github.com/kubecost/cost-analyzer-helm-chart/issues/1754#issuecomment-1335708206, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXPL5FCU6NXA74XTJ5GSUTWLJD4FANCNFSM6AAAAAARNEJ32U . You are receiving this because you commented.Message ID: @.***>

-- Sean Pomeroy Solution Engineer

Docs http://docs.kubecost.com/ | Blog https://blog.kubecost.com/ | Slack https://kubecost.slack.com/ | Github https://github.com/kubecost | kubecost.com

[Adam-Stack-PM]

Based on a chat with Kaelan.

Because this seems to be an issue w/ config and the Azure pricing endpoint, Ajay and Kaelan discussed having a PR for disabling other currencies besides USD and marking this as a proposed item for v1.100.

[kc-adawson]

Hey team, so is the fix on this for now to have users set the currency to US? Or is there no fix for this until 1.100? There are a few users who's pricing is affected by this issue still.

[AjayTripathy]

@kaelanspatel can you comment?

[kaelanspatel]

@kc-adawson Yes, from my exploration if they want their pricing to match the USD calculator it'll be to set their rate card config to USD.

I'm of the impression this is WAI on our end, and that if we want it other currencies to match that pricing calculator we'd probably want to contact microsoft support.

[kc-adawson]

Revisiting this issue - I have set up an Azure cluster with price history, and I've also integrated the Azure cost export

When adding the Azure France Central cluster into my federated setup, I get the following error message:

2023-02-04T17:27:21.294573202Z WRN could not fetch Azure node pricing for region francecentral from cache (will use default westus2 instead): file /var/configs/azurenodeprices-francecentral.json not found

I see no Azure "adjustments" under assets. I have this cluster available for debugging.

[AjayTripathy]

I would say that the node_total_hourly_cost is WAI, but if you're not seeing adjustments maybe @Sean-Holcomb is the right person to assign to.

[github-actions[bot]]

This issue has been marked as stale because it has been open for 360 days with no activity. Please remove the stale label or comment or this issue will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been inactive for 365 days with no activity.