Estimate Cost of New Relic with AWS data ingestion

[jimleroyer]

Description

As a (user), I need to be able to do (X) so that I can achieve (Y) outcome.

As a system ops of GCNotify and more specifically of NewRelic, I need the cost estimate of data transfer between AWS data center and NewRelic, Once we activate NewRelic features to track AWS events, So that I can weight in the cost benefit overall paired with a NewRelic plan switch.

WHY are we building?

We need it to weight in cost and benefits of using NewRelic. If we can switch to a compute cost, this will reduce our bill. We might want to leverage it further and find the right balance in terms of cost.

WHAT are we building?

• Estimate the cost from AWS to NewRelic data transfer. Offer different load of data transfer depending on the desired data and feature we could use in NewRelic.
• Look into the AWS marketplace for buying NewRelic from there and how much credits we would get for data transfer from AWS to NewRelic.

VALUE created by our solution

More leverage from NewRelic for its cost.

Acceptance Criteria

Given some context, when (X) action occurs, then (Y) outcome is achieved.

• [ ] Estimate the cost from AWS to NewRelic data transfer. Offer different load of data transfer depending on the desired data and feature we could use in NewRelic.
• [ ] Look into the AWS marketplace for buying NewRelic from there and how much credits we would get for data transfer from AWS to NewRelic.
• [ ] The cost analysis has been reviewed by dev lead.

QA Steps

No QA steps.

[ben851]

Costing is based on Firehose and Metrics streams.

Firehose sent to S3 in GB last 30 days: 26.5GB Firehose cost per GB: 0.029 Estimated Firehose Cost: $1 USD

Metric Streams: $0.003 per 1,000 metric updates I'm having trouble finding how many metric updates we do.

[ben851]

I think I can greatly reduce our cost by cleaning up the prometheus metrics. Will investigate.

[P0NDER0SA]

Ben is investigating linking NR to Staging AWS to see if it's easier to get an idea of the costing metrics

[P0NDER0SA]

Ben is still working on this in Staging before he can get these metrics

[ben851]

Managed to deploy to staging. Also deployed a change to kube state metrics to reduce the number of metrics sent to cloudwatch, which in turn has greatly reduced the metrics stream to new relic

[ben851]

Let new relic metric streams run overnight. Just merged a PR to turn it off

[ben851]

Confirmed no more metrics are being streamed. Will need to wait till end of month to see our NR bill to see how much this cost us

[sastels]

We can figure out the NR costs but still need the AWS costs

[sastels]

need to check egress data costs in AWS - Jimmy is investigating...

[P0NDER0SA]

We rolled back some of this work because it was causing minor residual issues since a couple of weeks ago. WE'll revisit this as a team next week.