feature: Cost Insights implementation

[niallthomson]

🔖 Feature description

Implementation of the Cost Insights plugin that more easily allows AWS users to view AWS resource costs. This would allow teams to view their associated AWS costs directly within Backstage and within the context of specific workloads.

Cost Insights is flexible with regards to the dimensions it can display. Expected dimensions that would be important to AWS users are breaking out costs by:

1. Total cost
2. Per service
3. Per account
4. Per region

It may also be necessary to allow a break down by arbitrary AWS tags on resources. For example if a user wants a breakdown "per environment" and these share a single AWS account.

Costs could be mapped to workloads and teams using two main mechanisms:

1. Cost allocation tags on resources
2. Account IDs where each workload gets its own AWS account(s)

🎤 Context

Research done:

https://github.com/backstage/backstage/blob/master/plugins/cost-insights/contrib/aws-cost-explorer-api.md

✌️ Possible Implementation

it does not look like it will be possible to provide a complete implementation for the CostInsightsApi as there are certain functions which likely cannot be opinionated about. Examples include:

• getAlerts : This could possible return information from Trusted Advisor etc. but for now this is not defined enough
• getDailyMetricData : It looks like this is intended to return "business metrics", which we cannot know for users

There are also functions which we can likely provide a default implementation for but are subjective to the users organization:

• getUserGroups : Returns the list of groups for a given user. By default we can look up the Group entities associated with the user via the catalog relationships.
• getGroupDailyCost : Returns the daily cost of a given group. Given the above by default we'll treat this as a Group entity and delegate to getCatalogEntityDailyCost

Due to this we will look to provide a minimal viable implementation which can be extended and overridden where necessary. Although this does not solve the entire problem it provides users with a solution that provides values with minimal work and can be extended if needed.

The implementation will primarily focus on getCatalogEntityDailyCost as this means the EntityCostInsightsContent component included with Cost Insights can provide entity level costs for anything in the catalog, which would include Component, Group, System, Domain and Resource (which is commonly used to model things like AWS accounts).

Mapping a given entity to cost information will be done using annotations. There are 3 main annotations proposed:

• aws.amazon.com/cost-insights-tags : Will filter costs using the given AWS cost allocations tags. This is expected to be useful for workload or team level costs where there is a consistent tagging strategy.
• aws.amazon.com/cost-insights-cost-categories : Will filter costs using the given AWS cost categories. This is expected to be useful for workload or team level costs where cost categories are used.
• aws.amazon.com/cost-insights-account-ids : Will filter costs using the given AWS account IDs. This is expected to be useful for AWS account level costs.

These annotations would be mutually exclusive on a single entity and we would only accept a single annotation meaning you cannot mix, for example, tags and cost categories. However you could use different annotations on different entities.

Example:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: example-website
  annotations:
    aws.amazon.com/cost-insights-tags: component=example-website
spec:
  [...]

It is expected that customers will have different priorities for what dimensions are used to group the data. For example some may only expect the "total cost" of an entity, but some may want to show cost by AWS account ID, region, AWS service etc. The tabs that show on the UI are driven by the data provided by the Cost Insights API implementation we would provide.

Its proposed to make these grouping dimensions configurable. Example:

aws:
  costInsights:
    tabs:
    - groupBy: # group this tab by AWS service
        key: DIMENSION
        value: SERVICE
    - groupBy: # group this tab by AWS account ID
        key: DIMENSION
        value: LINKED_ACCOUNT
    - groupBy: # group this tab by arbitrary cost allocation tag used by the customer
        key: TAG
        value: CostCenter

This reflects how the groupBy parameter of the Cost Explorer API works - Link.

The "total cost" would always be shown as the first tab.

The cost data can be retrieved in two main ways:

1. AWS Cost Explorer API - Not much initial setup but a cost per API call which will likely require caching
2. AWS Cost and Usage Reports queries - More initial setup but then less concern around on-going query volume to Athena

We'll provide separate implementations that can be switched out with configuration. Due to a simpler implementation initially we will only support the Cost Explorer API. Cost and Usage Reports support will follow if requested.

[gavinclarkeuk]

Might be of interest - we've created a suite of Cost Category rules in AWS to map costs to Backstage Components, Systems and Owners. The mappings are largely based on tags, but there are a few special case rules where AWS's tagging support isn't good. We regenerate the rules every night based on our backstage manifests, and we have a custom annotation in the manifest to configure any non-standard tag mappings.

Seems to me the simplest implementation of an AWS cost insights backend, would be to support both tags and categories that directly relate to backstage entities. Whilst it may not be sufficient for all use cases would be a straightforward v1 implementation. If categories are supported that leaves implementers with the option of writing more complex mapping rules in AWS using the category rules, avoiding the need for complex logic in the plugin.

I don't have a strong view on whether to use the cost explorer api or CUR reports, either would work for us. I guess cost explorer would have a lower barrier to entry, but might cost more in the long run.

[gavinclarkeuk]

Also would definitely need the ability to configure the cost aggregation used (unblended vs amortized) and other filters like charge type. For day to day reporting at a team/component level we only include the Usage related charge types and exclude things like Tax as that makes for easier comparisons and avoids some of the gaps in how different charge types are attributed to tags/categories particularly in CUR reports.

[niallthomson]

@gavinclarkeuk the Cost Categories are a good callout.

How do you map each entity to its Cost Category? You mentioned a custom annotation but it sounds like its for edge cases. Do you have a "standard tag mapping" that you can assume based on entity type/names?

[niallthomson]

Updated proposal with information regarding how the tabs that would be displayed could be configured.

[gavinclarkeuk]

How do you map each entity to its Cost Category? You mentioned a custom annotation but it sounds like its for edge cases. Do you have a "standard tag mapping" that you can assume based on entity type/names?

We have a standard tag that we put on all AWS resources. For most components the value will match the backstage component name, but we use the annotation for any custom mappings (e.g. where the tag doesn't match, or you have multiple tags mapping to one component).

We then have a nightly process that does an extract of the relevant data of the backstage manifests, a bit of data manipulation and some slightly mad terraform logic to craft the cost category rules with all the tag mappings and special case rules. The rules as we do it will only scale so far - I think the limit would be 1000 components. But the main point here is that you could use whatever logic you want in your category rules.

Whilst tags might be sufficient for a lot of people, the extra logic we can have in cost category rules helps us get a lot more under cost management. Some AWS services don't allow tagging or don't surface tags in the CUR reports, also we have a couple of sandbox/free for all AWS accounts that may not have all resources tagged, so we can add rules to handle them.

[niallthomson]

I have what might be a workable first version of this plugin in a branch. Documentation based on the proposed design is here https://github.com/awslabs/backstage-plugins-for-aws/tree/cost-insights-ce/plugins/cost-insights