ex.Redis - AWS implementation

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

[staycoolcall911]

Keep

[hasanaburayyan]

This implementation will be tricker because AWS Elasticache appears to needs a VPC for the cluster to be connected to.

See: https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/VPCs.EC.html

We currently do not have any resources in AWS that require a VPC and this introduces a few new concerns into the SDK.

1. We need a VPC (either create one or use existing)
2. If we want to have a function access the cluster then the functions must be deployed into the VPC with proper subnet and security groups

Some possible paths forward:

1. A potential implementation is that we use (lazy initialization) to create an app wide private VPC which is only created when a resource that needs it is instantiated. That VPC will be used to deployed Elasticache clusters and functions that require network access to the redis
2. We ask users to pass in an existing VPC to be used for these resources
3. Investigate a temporary alternative for providing cache in AWS

love to get some feedback on implementation.

Really want to focus on what the best DX is going to be, I feel like its option 1 or 3

[Chriscbr]

Great overview Hasan. I'll share some thoughts but I should start with a huge disclaimer that I don't have a lot of experience working with VPCs. 😅

For option (1) I wonder if that means all cloud.Function's in a Wing app would have to get created by the VPC? Or only those that use the Redis resource? If lambda A is in the VPC and performs operations on the cache, can lambda B outside of the VPC call invoke on the lambda inside the VPC? What if I want a lambda to call operations on Redis and also to make requests to the internet, is that possible? Alternatively, are there any downsides to putting everything in a VPC?

I'm also curious how important others feels a Redis / Cache resource is for beta, or if the current Table and Bucket resources we have working both on simulator and AWS are enough to provide as a starting point.

I think we should decide as a team on whether VPCs are a "first class constraint/option" supported by the SDK -- that is, configuring whether each resource is in a VPC or not, and/or automatically placing resources in a VPC where necessary. We know Wing gives users full flexibility to create resources that correspond 1:1 to the terraform providers (so they can directly create VPCs etc.), and use Wing plugins to apply customizations to existing resources. But does adding this new degree of freedom unlock new capabilities for Wing users, and what is the maintenance cost (for users and for us)? This might impose costs, like we may need to run integration tests once with apps in VPCs and once outside of VPCs to check that apps work correctly in both situations.

My initial feeling is that if VPCs will unlock key use cases for the devs building greenfield projects with Wing, or not including it in the SDK would be a blocker to a large enough fraction of Wing's target audience, then it's probably worth standardizing and supporting them in the SDK. Otherwise, maybe supporting VPCs is something we can enable out of band as a community-supported effort with the Wing library ecosystem, and we can pursue option (3) instead.

[hasanaburayyan]

@Chriscbr l agree I think we should discuss how the SDK supports VPCs. As for VPCs I know more than I would like to about them 😂 so let me try to answer some of the questions.

I wonder if that means all cloud.Function's in a Wing app would have to get created by the VPC?

Only the lambdas that need to access the cache would have to be behind the VPC

If lambda A is in the VPC and performs operations on the cache, can lambda B outside of the VPC call invoke on the lambda inside the VPC?

Yes the lambda invocation is through the AWS API so as long as lambda B has access to the internet and can reach AWS API then it works.

What if I want a lambda to call operations on Redis and also to make requests to the internet, is that possible?

This is possible if the VPC is configured with internet egress. Its pretty common and easy to setup, requires an internet gateway to be created with the VPC. (there is an exception here for things that can be done with private links but thats another topic)

[ekeren]

From a wing user perspective, what is the difference between the fact that on aws a cloud.Function requires a Bucket to store the code in to the situation that Elasticache requires to run on a VPC? (I understand the difference in complexity)

[hasanaburayyan]

@ekeren I dont think its something the use should have to care about. Which makes me think that something like option 1 makes the best DX. If users need to specify the VPC then that is something they can do with plugins.

[ekeren]

How hard is doing number 1?

[hasanaburayyan]

Ill take a look this weekend, the biggest question mark for me is will it be difficult to bind lambda functions to a VPC when they are using the cache. Other than that I think its mostly straight forward 1 - 2 days. We already adopted this sort of app wide resource sharing model with Azure where we needed to share resource groups and storage accounts.

This just has the added complexity of networking fun. Basically we could add the VPC that comes with a single private subnet and internet gateway (with a route table entry). Then it is probably easiest to set up a single security group that allows communication with anything in the same SG. Id steer clear of multi-az and such for now (leave that for plugins).

[hasanaburayyan]

@ekeren following up on here to your question. The answer is its not hard, I was able to prove it all out with just a bit of work. Basically we can just add a vpc to lambda function during the redis _bind() step.

The biggest con to this solution is that keeping this cluster behind a private subnet will require a NAT gateway for internet egress. (Only 1 the whole app shares the VPC's config) I think a managed NAT is something like $15/month

tagging @Chriscbr as well

Bellow are the screenshots from test.

Example wing code:

bring cloud;
bring redis;

let r = new redis.Redis();
let r2 = new redis.Redis() as "r2";

new cloud.Function(inflight (s:str):str => {
   print("${r.url()}");
   print("${r2.url()}");
   let connection = r.ioredis();
   let connection2 = r.ioredis();
   connection.set("wing", "does redis");
   let value = connection.get("wing");
   print("${value}");
   connection2.set("wing", "does redis again");
   let value2 = connection2.get("wing");
   print("${value2}");
}) as "test";

Lambda VPC Config:

Redis Clusters:

Cluster Networking:

Tiny VPC Topology:

Successful Lambda Test:

[Chriscbr]

@hasanaburayyan cool to see it working!

Basically we can just add a vpc to lambda function during the redis _bind() step.

To check my understanding, does this mean no changes needed to be made to other resources (like cloud.Function or App) in order to make it work?

My only concerns were if (1) introducing redis required some cross-cutting changes (e.g. adding a "in_vpc" flag to any resources), or (2) if any changes performed during _bind caused functional behavior of inflight code to change. (1) is a risk just from a maintenance perspective, and (2) is a risk since it would be a leak of operational concerns into userland.

(For example, let's say a user has a cloud.Function whose inflight calls some endpoints on the internet and puts some messages into a queue. Suppose that once a user more code to the inflight to call r.ioredis(), then suddenly the previous lines in the inflight started failing because the cloud.Function was moved inside a VPC. That would be pretty surprising as a user... 😬 Based on this I think we'll probably need to guarantee that cloud.Function behaves the same inside and outside any auto-generated VPCs).

If we can avoid all of those then it sounds like a huge win! 👍

[hasanaburayyan]

Spoke with @Chriscbr but want to share notes here in thread.

For example, let's say a user has a cloud.Function whose inflight calls some endpoints on the internet and puts some messages into a queue. Suppose that once a user more code to the inflight to call r.ioredis(), then suddenly the previous lines in the inflight started failing because the cloud.Function was moved inside a VPC.

The access to the public API or other internet resources the lambda were calling will not be disconnected since the VPC's private network is setup to use a NAT gateway and Internet Gateway. So anything in the VPC's private network still has access to calling things outside of the VPC. As for other resources that need to invoke the lambda, even though the lambda is now hidden behind a private VPC the lambda invoke API is done through the AWS API and does not require network connectivity. So we should not have to worry about this model breaking that connectivity for lambdas. note: there may be other resources in the future that is not true of, but I cannot think of any off the top of my head.

introducing redis required some cross-cutting changes (e.g. adding a "in_vpc" flag to any resources),

Wing users would not be required to provide any in_vpc flag, the fact that there is a vpc involved at all is completely removed from the user. This is only something that would be seen after the resources are deployed. As well there is the implication of using a managed NAT gateway that will run the user about $30/month

[monadabot]

Congrats! :rocket: This was released in Wing 0.8.34.