search

open-mpic / aws-lambda-python

An implementation of the Open MPIC API using AWS-Lambda serverless fucntions written in Python as well as AWS API Gateway.
MIT License
7 stars 4 forks source link

AWS resources actually needed to run Open-MPIC #20

Open defacto64 opened 1 week ago

defacto64 commented 1 week ago

Hello,

we have been testing open-mpic for a while, now. Starting from the 10 sep commit, we noticed that deploying open-mpic on AWS requires a number of "extra" resources that have a significant impact on costs (compared to the cost of the "lambda" resource alone which is quite modest), therefore we would like to understand if those resources are actually all necessary or not. In particular, we'd like to understand what is the resource that AWS charges under the EC2-Other item (see chart below) and if it is possible to reduce its consumption; that resource accounts for 99% of the overall cost. If it's necessary, so be it - but we'd like to understand this better.

immagine

I apologize if this question is trivial, but in fact I personally don't know much about AWS and I'm not entirely clear on the mapping between the required AWS resources and the technical-functional-operational requirements of open-mpic.

gcimaszewski commented 1 week ago

Hi @defacto64 , thanks for your help in testing and debugging Open MPIC. There are a number of things that can fall under the "EC2-other" category (including elastic IP addresses and NAT gateway charges) , but this is still surprising. If you go to Cost Explorer, filter by the "EC2-Other" service, and group by dimension "API operation" or "Usage type," you'll have a more granular view of what is being billed. (It would also be helpful for us and others to share those results here.)

defacto64 commented 1 week ago

Thank you for replying @gcimaszewski . By doing as you suggest, it turns out that the "offending" resource is NatGateway. Do you believe it is expected? There may be a way to reduce significantly its consumption?

sciros commented 1 week ago

@defacto64 this is a "known issue" that we have run into due to how we are currently enabling DNSSEC validation (which requires Route 53 in AWS if we are sticking with AWS Lambda; the "default" DNS resolver that Lambda (or GCF for that matter) leverages does not support DNSSEC). There is a fairly high running cost associated with NatGateway so your finding corroborates our own.

This has prompted us to look at a few alternatives for how we might architect the deployment configuration of an open-mpic implementation. We are currently exploring a containerized approach that would enable us to not rely on Route 53 for DNS lookups, and furthermore may enable us to build a more hyperscaler-agnostic implementation.

In the meantime, this is nothing you're doing wrong but merely how the pricing works around the resources that the current configuration requires. We're hoping to address this because high fixed cloud costs can be a barrier to adoption and the goal of an open-mpic implementation is of course to encourage and enable adoption.