search

finopsfoundation / focus_converters

Parent repository to hold all common documentation and code samples for all FOCUS Converter projects
MIT License
77 stars 43 forks source link

Azure EA conversion fails: 'BillingCurrencyCode, CostInBillingCurrency' not found in data #304

Open sxdjt opened 8 months ago

sxdjt commented 8 months ago

Describe the bug

After downloading an Azure EA detailed billing file and running the converter, it fails saying BillingCurrencyCode, CostInBillingCurrency' not found in data.

To Reproduce Steps to reproduce the behavior:

  1. Download Azure EA detailed billing file 
    Cost Management > Usage + Charges > Download > Amortized Charges
  2. Run: 
    # focus-converter convert --provider azure --data-path Detail_Enrollment.csv --data-format csv  --export-path .
  3. Error: 
    ValueError: Column(s) 'BillingCurrencyCode, CostInBillingCurrency' not found in data

Expected behavior The converter runs as advertised.

Screenshots See attached files containing detailed error message and the headers of the downloaded billing file.

Desktop (please complete the following information):

csv_headers.txt error_out.txt

csyvenkyteck commented 8 months ago

Hi @sxdjt,

We had a working group call today, two ppl on the call confirmed that out-of-the-box Azure Export datasets have both of these fields. I feel this is more likely connected to converter's validation logic. There is a sample file in the repo, I'd recommend checking out the validator --help flag to see the argument that can help you pass in the sample data for testing.

sxdjt commented 8 months ago

Hello @csyvenkyteck - I downloaded the sample file and the conversion ran as expected.

I'm not sure why I am missing those fields in the exported data. Is there a difference between exporting from Azure Portal vs. using the cost management API? I can't imagine there would be, but...

MmmmJoel commented 8 months ago

I confirm that in our org, those two fields do not exist in the raw billing exports. (EA agreement, US-based).

csyvenky-finops commented 8 months ago

@sxdjt, @MmmmJoel, I'm unsure what else I can add. My previous posts assumed you are using the Azure Cost Mgmt. Export facility on an EA account type.

Just to verify, today I called the following APIs and can confirm yet again these fields exist in the output:

https://management.azure.com/providers/Microsoft.Billing/billingAccounts/{{billingAccount}}/providers/Microsoft.CostManagement/generateCostDetailsReport?api-version=2023-03-01

https://management.azure.com/providers/Microsoft.Billing/billingAccounts/{{billingAccount}}/providers/Microsoft.CostManagement/costDetailsOperationResults/?api-version=2023-03-01

sxdjt commented 8 months ago

@csyvenky-finops I think that is the issue; exporting the data from Azure Portal gives different results that using the API. Not sure if Microsoft intended that or not...