Data transforms in FinOps hubs are a critical part of improving the data quality and filling gaps to facilitate FinOps operations. There are a number of issues and open questions that have come up over time about the data coming from various systems. This issue is used to track requested changes, ideas for consideration, and any open questions that need an answer.
🛠️ Solution
### HubScopes
- [ ] Detect any scopes that are duplicated via the drop-by folder path
- [ ] Create an array of missing data based on the configured retention setting
- [ ] Add min/max data timestamps for each scope
### CommitmentDiscountUsage (reservation details)
- [ ] What are these columns: ReservedHours, TotalReservedQuantity, UsedHours
- [ ] Is SkuName the same as x_SkuSize?
- [ ] What are the possible values for Kind?
- [ ] Consider adding ServiceSubcategory (1.1)
- [ ] Does this dataset include non-hour units (namely the ReservedHours column)?
### Costs (FOCUS cost)
- [ ] Add ServiceModel
- [ ] Add CommitmentDiscountSpend/UsageEligibility
- [ ] Consider fixing PricingQuantity and ContractedCost with bad values due to the wrong PricingQuantity scale
- [ ] Consider fixing "Other" ServiceCategory values
- [ ] Consider replacing "Microsoft Corporation" PublisherName values
- [ ] Fix FOCUS conformance gaps
- [ ] ContractedCost/UnitPrice are 0 for commitment discount purchases
- [ ] Savings plan purchases show a ChargeFrequency of "Usage-Based" but should be "One-Time" or "Recurring"
- [ ] Savings plan purchases do not have a ServiceName
- [ ] x_BilledUnitPrice is 0 for commitment discount purchases
- [ ] Lowercase Microsoft 1.0-preview ResourceId values
- [ ] Should commitment discount purchases have a time for ChargeperiodStart/End?
- [ ] Should Adjustment costs be on List/ContractedCost?
- [ ] PublisherName "Microsoft Corporation" -> "Microsoft"
- [ ] File a bug on CM for ContractedCost being off by the x_PricingBlockSize
- [ ] PricingUnit is incorrect for reservation purchases
- [ ] Reservation usage has List/ContractedCost but Effective/BilledCost == 0
- [ ] Create a pattern for identifying (and filtering out) commitment discount purchases (so they aren't dobule-counted)
- [ ] PricingUnit is "Units" when it should be "Operations"
- [ ] Consider setting BillingAccountType/SubAccountType for non-Azure datasets
- [ ] Replace SkuId/SkuPriceId for Microsoft EA rows
- [ ] Add SkuPriceIdv2 for Microsoft rows
- [ ] Add warnings for any changes applied to the data
- [ ] Add warnings (errors?) when there's a difference between meter metadata in cost and price datasets
- [ ] Create alerts when there are differences between cost/price data (???)
- [ ] Confirm the ProviderName value for AWS and GCP data and update it in the ProviderName backfill
- [ ] What should the ListCost/UnitPrice and ContractedCost/UnitPrice be for rounding adjustments?
- [ ] What should the ListCost/UnitPrice and ContractedCost/UnitPrice be for commitment discount purchases?
- [ ] Why do price sheet savings plan usage prices not match cost effective prices?
- [ ] Consider adding a unique charge ID per row
- [ ] DQ: Some purchases (e.g., M365) have ChargeFrequency = "Usage-Based"
### Prices (price sheet)
- [ ] Handle null MeterId
- [ ] Consider adding x_CommitmentDiscountMinimum (as in the minimum commitment amount; e.g., 100TB, 1PB, 10PB for storage)
- [ ] Consider adding x_CommitmentDiscountNormalizedRatio
- [ ] Consider adding x_ServiceModel
- [ ] Consider cleaning up x_SkuRegion
- [ ] Consider creating new rows for reservation usage
- [ ] SkuPriceId should not be "_" for unused savings plan rows
- [ ] Why is MeterName missing for some records (e.g., reservations)?
- [ ] Why are 309 meters missing all meter metadata for MCA?
- [ ] Why is MarketPrice not the on-demand list price for savings plan prices?
- [ ] Why is MarketPrice ~= UnitPrice (but not ==) for savings plan prices?
- [ ] Why are there Az Stack and Az Comm Services meters with "Unassigned" ProductName, MeterCategory, MeterSubcategory, MeterName, MeterRegion?
- [ ] Does ListUnitPrice/x_EffectiveUnitPrice match usage data for savings plans?
- [ ] Does x_SkuMeterName match usage data?
- [ ] Does the effective start date for reservation prices change based on the export date?
- [ ] How can we differentiate between Microsoft and third-party meters?
- [ ] How do we create a unique ID for third-party meters for MCA?
- [ ] How do we differentiate dev/test for MCA?
- [ ] Compare join vs. lookup performance for populating savings plan prices
- [ ] Compare join vs. lookup performance for populating pricing unit columns
- [ ] Compare join vs. current per-row approach for populating commitment discount eligibility
### Recommendations (reservation recommendations)
- [ ] Move ProviderName upstream
- [ ] Why is SubscriptionId not included?
- [ ] Add support for ingesting multiple types of recommendations (not a transform issue)
- [ ] Add Advisor recommendation columns
### Transactions (reservation transactions)
- [ ] Consider looking up the billing account ID from cost data to fix the invalid BillingAccountId column
- [ ] Is the Currency column the pricing or billing currency?
- [ ] A "Cancel" ChargeClass is not valid in FOCUS -- should we track this differently or leave it as an acceptable conformance gap?
- [ ] Move ProviderName upstream
- [ ] Map BillingFrequency values to ChargeFrequency
- [ ] Should we include ChargePeriodEnd?
🙋♀️ Ask for the community
We could use your help:
Please vote this issue up (👍) to prioritize it.
Are there any open questions or concerns you have about the Cost Management data? How can we improve it?
⚠️ Problem
Data transforms in FinOps hubs are a critical part of improving the data quality and filling gaps to facilitate FinOps operations. There are a number of issues and open questions that have come up over time about the data coming from various systems. This issue is used to track requested changes, ideas for consideration, and any open questions that need an answer.
🛠️ Solution
🙋♀️ Ask for the community
We could use your help: