CostJanitor: List cost for a Capabillity/AWS account

SEQUOIIA commented 4 years ago

Use billing API to acquire total costs if AWS account is attached to Capability

More research needs to be done. Is there even a nice way of doing this using the existing AWS API?

[x] Research: Discover how to acquire total costs of an AWS account. Potential leads: API?
[x] Implement host process for listening on Capability events
[x] Implement host process converter to map capability events to aggregates
[x] Implement logic to map aggregates to commands
[x] Implement AwsCostClient
[x] Implement AwsCostClient DTOs
[x] Update domain model to be more aligned with AWS model
[x] Implement DTO to Aggregate converters for AwsCostClient
[x] Implement host process for API controllers
[x] Implement report controller
[x] Implement required view models
[x] Enable Microsoft Identity middleware
[x] Create topics (stretch)
[ ] Implement Domain & Integration eventhandlers (stretch)
[x] Implement KafkaProducer (stretch)
[x] Test janitor

It is possible to use either the AWS SDK or the AWS v2 CLI to query a capability for various cost metrics. (yes, both work, move this to comment section)

Using the SDK & API: https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html

(move to comment section)

Using AWS CLI: https://docs.aws.amazon.com/cli/latest/reference/ce/get-cost-and-usage.html

A decision is required here to select a path of implementing this feature.

Current working repo: https://github.com/dfds/cost-janitor

Aha! Link: https://dfds.aha.io/features/CLOUD-103

SEQUOIIA commented 3 years ago

Current state as of Mon 11 Jan 2020 09:32 PM:

CI/CD pipeline for Cost Janitor is up and running
Dockerfile that builds project has been created
Kubernetes manifest files for the project has been created
A prototype HTTP server written in Go with a route that takes AWS account id and returns current total costs for the month. This endpoint is also protected with OIDC, using the same client id and configuration as for Capability Service at the moment. To be changed.
All of this is currently deployed on Hellman in my sandbox namespace sandbox-emcla-pmyxn

Endpoint: https://api.hellman.oxygen.dfds.cloud/cost-janitor/get-monthly-total-cost/AWS-ACCOUNT-ID

e.g. for dfds-log AWS account it would be https://api.hellman.oxygen.dfds.cloud/cost-janitor/get-monthly-total-cost/736359295931

Then it expects a Authorization header with "Bearer TOKEN" with TOKEN being replaced with a base64-encoded jwt token.

Can be seen in https://github.com/dfds/cost-janitor/blob/master/src/main.go

SEQUOIIA commented 3 years ago

I took at look at the API pricing for CE and did some napkin math:

1 Request = 0.01 USD Current amount of Capability contexts = 118

Assuming each context/AWS account was queried once every hour for 24 hours: 1 hour: (118 0.01) = 1.18 USD 24 hours: 1.18 24 = 28.32 USD

Now, if that was done once every hour for 30 days: 30 days: 28.32 * 30 = 847 USD

Seems like a lot of money for getting to know how much money we spend.

Seems like a reasonable option would be to whenever an account is queried, cache the result for X amount of time(1 day would mean with the current amount of accounts something along the numbers of 35.4 USD a month).

Suggestions, thoughts? @ZaradarDFDS @carlsendk

SEQUOIIA commented 3 years ago

Current state as of Mon 11 Jan 2020 09:32 PM:
* CI/CD pipeline for Cost Janitor is up and running

* Dockerfile that builds project has been created

* Kubernetes manifest files for the project has been created

* A prototype HTTP server written in Go with a route that takes AWS account id and returns current total costs for the month. This endpoint is also protected with OIDC, using the same client id and configuration as for Capability Service at the moment. To be changed.

* All of this is currently deployed on Hellman in my sandbox namespace _sandbox-emcla-pmyxn_
Endpoint: https://api.hellman.oxygen.dfds.cloud/cost-janitor/get-monthly-total-cost/AWS-ACCOUNT-ID

e.g. for dfds-log AWS account it would be https://api.hellman.oxygen.dfds.cloud/cost-janitor/get-monthly-total-cost/736359295931

Then it expects a Authorization header with "Bearer TOKEN" with TOKEN being replaced with a base64-encoded jwt token.

Can be seen in https://github.com/dfds/cost-janitor/blob/master/src/main.go

Current state as of Tue 12 Jan 2020 3:44 PM:

All of the previous
Date is no longer hardcoded, but rather dynamically selects the current month
For easier testing with services that doesn't support our OAuth setup, an endpoint with Basic has been set up.
Results from AWS are cached for an hour, cache resides in Redis

dfds / backstage_originaldfdsfork

CostJanitor: List cost for a Capabillity/AWS account #81