search

FinOps-Open-Cost-and-Usage-Spec / FOCUS_Spec

The Unifying Specification for Cloud Billing Data
https://focus.finops.org
Other
184 stars 39 forks source link

[Work_Item] Add SKU categorization hierarchy #608

Open flanakin opened 1 month ago

flanakin commented 1 month ago

1. Problem Statement *

Describe the problem, issue, use case, or opportunity that this work item addresses. Include practitioner quotes illustrating real examples a) of questions being asked by practitioners and b) value unlocked by answering these questions, if available.

  • What is the problem?: Explain the context and why it needs resolution.
  • Impact: Describe how the problem affects users, systems, or the project.

The most fundamental thing we are describing in FOCUS is the SKU. Yet as-is, the only thing we know about the thing we're explicitly being charged for is 2 IDs that need to be looked up in another system. And how those IDs map to the provider's price list is not clear since each provider uses different terms for those attributes. This makes relying on SkuId and SkuPriceId very difficult.

Use cases:

2. Objective *

State the objective of this work item. What outcome is expected?

  • Success Criteria: Define how success will be measured (e.g. metrics and KPIs).

Since SKU attributes are the most common way leaders throughout an organization look at their cloud usage, there needs to be a human-friendly categorization of SKUs. At a bare minimum, we need a SkuCategory that is the same as ServiceCategory (since there will be a one-to-one mapping in some cases). A SkuSubcategory would also be helpful, but having a normalized column for this might require a significant effort.

Beyond normalized columns, FOCUS needs to allow practitioners to categorize their SKUs. Each of the main providers have a logical way to organize SKUs already so this is mainly about identifying clear names to describe a taxonomy of products. As an example, AWS has product_group and product_family in the CUR, GCP has SKU group as a concept outside of the cost data, and Microsoft has MeterCategory and MeterSubcategory. In some cases, providers merge multiple concepts into these attributes because there isn't a very well-defined structure that can support such a broad catalog of services. Having a predefined structure that is more open would go a long way to making it easier to understand what SKUs are being used across an expansive account, especially when those SKUs span services and need to be measured for things like commitment discounts.

There's already a somewhat common taxonomy for product categorization that exists that we could leverage as a starting point. You can find many different websites that discuss product categorization, but as a starting point, see https://catsy.com/log/product-categorization.

3. Supporting Documentation *

Include links to supporting documents such as:

  • Data Examples: [Link to data or relevant files; DO NOT share proprietary information]
  • Related Use Cases or Discussion Documents: [Link to discussion]
  • PRs or Other References: [Link to relevant references]

See Microsoft ServiceFamily, MeterCategory, MeterSubcategory.

Related to #315

SkuCategory would likely mimic ServiceCategory, since every usage and purchase charge would map to a SKU. For instance:

SkuSubcategory may differ from ServiceSubcategory. This would need more discussion.

For examples of the provider-specific SKU hierarchy, refer to the proposed SKU categorization appendix in the potato branch.

4. Proposed Solution / Approach

Outline any proposed solutions, approaches, or potential paths forward. Do not submit detailed solutions; please keep suggestions high-level.

  • Initial Ideas: Describe potential solution paths, tools, or technologies.
  • Considerations: Include any constraints, dependencies, or risks.
  • Feasibility: Include any information that helps quantify feasibility, such as perceived level of effort to augment the spec, or existing fields in current data generator exports.
  • Benchmarks: Are there established best practices for solving this problem available to practitioners today (e.g. mappings from existing CSP exports that are widely used)?

Add normalized and provider-specific columns for SKU categorization. Normalized columns would be similar to ServiceCategory and ServiceSubcategory but provider-specific columns would need to support more levels to account for the variations in types of related products. Note this is not a "type", but a grouping. A "type" is an attribute of a thing and not the grouping those "types" are categorized into.

5. Epic or Theme Association

This section will be completed by the Maintainers.

  • Epic: [Epic Name]
  • Theme: [Theme Name, if applicable]

TBD

6. Stakeholders *

List the main stakeholders for this issue.

  • Primary Stakeholder: [Name/Role]
  • Other Involved Parties: [Names/Roles]

TBD

shawnalpay commented 1 month ago

@shawnalpay to port to Work Item issue template.

jpradocueva commented 1 month ago

Summary Members' call on Oct 17:

#608 [FEEDBACK]: SKU Categorization Columns Primary Issue: Practitioners have requested SKU categorization columns to help organize and differentiate various SKUs at different levels of granularity. Core Problem: The current specification lacks sufficient categorization for SKUs, making it difficult to analyze or group SKUs effectively, especially in large multi-cloud environments. This is particularly important for normalized data as well as provider-specific attributes. Divergent Views: Some members favored a simple categorization schema, while others argued for more granular, provider-specific categories. There were also discussions about maintaining consistency while supporting varied provider taxonomies. Final Agreement: The group agreed to move forward with both normalized and provider-specific SKU categorization columns. This will help practitioners categorize and analyze SKUs across different cloud providers while maintaining provider-specific details where necessary. Action Items:

shawnalpay commented 1 month ago

@flanakin Per our discussion last week, I've converted this issue into the Work Item format! A couple things:

shawnalpay commented 1 month ago

I'm going to add back the needs examples label because it will be very helpful to our discussion with stakeholders (and amongst ourselves) if we can see some real-world sample data.

Here's just one example. GCP SKU 026-B8EB-D1EF has a multi-level product taxonomy of: GCP -> Compute -> Hyperdisk -> Hyperdisk Storage Pools -> Balanced -> IOPS -> Standard

Having a spreadsheet of many such examples (it doesn't have to be a full dump of all SKUs; say at least 1000?) will help drive understanding and prioritization.

robmartin33 commented 3 weeks ago

Presented and discussed in the 11/1 FUG NA Meeting. Interest from attendees Eli Lilly, Nissan Americas, both adopters, in getting this breakdown of category by meter/service Type.

ijurica commented 3 weeks ago

@flanakin @shawnalpay Is this the link to the blog that was suggested as a starting point? https://catsy.com/blog/product-categorization/

jpradocueva commented 2 weeks ago

Notes from the Maintainers' call on November 4:

Context:The addition of a SKU taxonomy would help standardize how SKUs are categorized across cloud services, improving resource grouping and cost tracking. This taxonomy must balance the need for detail with general applicability across diverse provider SKUs. Level of Effort Required: Very High — Creating a universal SKU taxonomy is complex and requires input from multiple stakeholders to ensure it accommodates varied provider structures.

jpradocueva commented 2 weeks ago

Comments from Members' call on November 7:

#608: The group analyzed this issue, and focused on SKU categorization, hierarchy, and taxonomy. TF-3 prioritized SKU ID as the most critical element for the current release.

jpradocueva commented 5 days ago

Action Items from TF-3, Nov 8: