VPC lambda support

[siegesmund]

I need up to create its function inside a VPC. It's easy to do with Apex, and I've tried using the Apex style configuration, but my functions are still being deployed without a VPC configuration. Is there a way to do this?

[tj]

Ahh not yet, I don't think it's possible to deploy API Gateway to a VPC right now (only Lambdas), but keep an eye on #14

[siegesmund]

I'm not trying to put the API Gateway inside the VPC; #14 is a different question. The goal is to put the lambda in the VPC, so it can access resources like an RDS instance that's only accessible inside the VPC. It can still function as a handler for the API Gateway endpoint(s). This is a feature of the Serverless framework, configurable in serverless.yml.

Perhaps my initial question was a little inarticulate. Does this make sense? And is it possible/or on the roadmap? And BTW, thanks TJ :)

[tj]

ahh ok cool thanks, I'll reopen! I definitely wouldn't mind getting that in there

---

• [ ] implement
• [ ] documentation (listing subnet ids etc, link to AWS docs)
• [ ] QA with and without

[siegesmund]

Not sure the complexity of this one, but If you can point me in the right direction, I'd be happy to see if I can't make the changes in an pull request.

[choonkeat]

I'll be happy if I can use the same json as apex in my up.json

{
  "vpc": {
    "name": "vpc-...",
    "securityGroups": ["..."],
    "subnets": ["subnet-..."]
  }
}

[vietbui]

It's a deal breaker as in lots of user cases, the lambda function needs access to VPC resources (e.g. RDS, internal load balancer...). One option is to change back to deploying the service in ECS and access via public load balancer but it's an overhead and I really enjoyed using up so far.

By the way, thanks for your great work TJ.

[a-golovanov]

@vietbui as a temporary workaround you can manually specify VPC for the already created lambda and it will stay configured across deployments.

[vietbui]

Nice workaround @a-golovanov! Thank you.

[vespertilian]

Just FYI for anyone else following @a-golovanov temporary workaround. I had to add the extra credentials from this page of Apex before deploys would work for me:

https://github.com/apex/apex/blob/master/docs/aws-credentials.md

[cubabit]

Does anyone know if you have to open up any ports in the security group as I get this error in up logs:

  01:58:53pm FATA error: initializing relay: waiting for http://127.0.0.1:46113 to be in listening state: timed out after 15s

I'm not sure what is trying to connect to what and as the error is from the logs of the lambda function and its using the localhost ip I'm not sure if it is a security group issue

[naartjie]

Hi @cubabit, I don't have too much context or experience using apex/up, but I didn't have to set up anything extra to get the logs (only to access RDS, that required a manual step on my part).

Looking at your error, is that what you get invoking up logs in your terminal? I would venture to say that might be your actual lambda log output. Have you tried going to CloudWatch and looking at the logs there to confirm that's not the case? For example if your app is not listening on port number in process.env.PORT i.e. 46113 in your case, that could be the problem. Just thinking out loud though, I could be wrong.

[cubabit]

Thanks @naartjie for replying.

Turns out my app was not listening on process.env.PORT like it should, but instead on another differently configured port. Once I changed it to use that env var everything worked!

[keshavab]

+1. It would be awesome to have support for deploying lambda in given VPC and subnet since we would have backend resources accessible only within VPC.

[simap]

+1 adding the lambda to a vpc didn't work for me, seemed to prevent it from reading env vars. would rather not make my RDS public.

[cubabit]

I can see why this is probably not going to be a priority because web apps in Lambda experience a huge performance penalty if they are in a VPC (see this article).

Apex supports VPC, as it is aimed at non-user facing functions, unlike Up.

[simap]

@cubabit and that may or may not kill my use-case assuming warming doesn't work. If there was a way to lock down a public rds, I'm game to try that. Sounds like eventually aws will fix that performance problem, or serverless Aurora becomes generally available presumably with some kind of iam role-based access. It could be up isn't the best fit for this particular app at this time.

[cubabit]

@simap You can lock down RDS to the IP range of AWS Lambda IPs, which is a better than nothing solution but obviously not great for information security.

http://blog.rowanudell.com/updating-security-groups-with-lambda/

Also, the problem with warming is it is not always effective. Lambda may according to traffic scale up and create a new instance to handle your request at any time, which will be cold. API Gateway will dispatch to that instance and therefore suffer from a slow response time.

[simap]

@cubabit thanks for those links, thats close to what I was looking for and at least reduces the attack vector.

I'm not sure if the performance problem is there anymore. I just set up a basic lambda in a vpc consuming an simple private http endpoint on an ec2 and the performance looks great. First curl was ~.5s. cold load test w/ 200 connections:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   695.33ms    1.23s    9.73s    85.82%
    Req/Sec   125.19     53.29   240.00     70.13%
  Latency Distribution
     50%  103.71ms
     75%  687.64ms
     90%    2.48s 
     99%    5.46s 
  10402 requests in 10.10s, 4.72MB read
Requests/sec:   1029.74
Transfer/sec:    478.67KB

then warm:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   117.38ms   93.80ms   3.65s    98.62%
    Req/Sec   159.94     38.36   280.00     76.45%
  Latency Distribution
     50%  108.62ms
     75%  124.30ms
     90%  152.60ms
     99%  228.24ms
  15464 requests in 10.06s, 7.02MB read
Requests/sec:   1537.87
Transfer/sec:    714.87KB

lambda code:

var http = require('http');
exports.handler = function(event, context) {
  http.get("http://172.31.29.158:8080/test", function(res) {
    let resp =  {
        statusCode: 200,
        body: "hello world"
    }
    context.succeed(resp);
  }).on('error', function(e) {
    context.done(null, 'FAILURE');
  });
}

the internal endpoint is just http-server with a tiny static file. to compare I have the basic hello world lambda not in a VPC.

var http = require('http');
exports.handler = function(event, context) {
  let resp =  {
      statusCode: 200,
      body: "hello world4"
  }
  context.succeed(resp);
}

first curl is also .5s and benchmark is not that much better considering it has 1 less hop to go through to service the request:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   381.86ms  834.26ms   5.44s    90.13%
    Req/Sec   200.44     74.42   373.00     69.00%
  Latency Distribution
     50%   67.13ms
     75%  115.36ms
     90%    1.20s 
     99%    4.26s 
  18642 requests in 10.06s, 8.46MB read
Requests/sec:   1853.30
Transfer/sec:    861.49KB

[dusty]

I just manually added VPC support to a test app and I just had to attach this policy to the role to make it work. AWSLambdaVPCAccessExecutionRole

When I first tried to do it, the Save button was going from a gray disabled state and returning back to orange, without any error notification. I had to inspect the ajax payload in chrome to realize it wanted to tell me I didn't have the VPC role, then went googling.

[arxpoetica]

I tried doing a manual VPC on the Lambda, which worked great, but when I upped some changes to the Lambda again it overwrote all my VPC changes. 😬😩

[arxpoetica]

by the by, here's a handy guide: https://docs.aws.amazon.com/lambda/latest/dg/vpc.html

[tikotzky]

Are you sure you'd need a network load balancer in order to put a lambda in a VPC?

I have a few lambda functions managed by apex currently running in a VPC. Im put them in a VPC to

1. have their outbound traffic egress via a static IP
2. Give the access to resources available only on the VPC All I had to do to set it up was edit the VPC setting on the lambda function in the AWS interface.

I am then able to continue deploying via up and the VPC configuration is left untouched...

[dusty]

Me too. Here is what I had to do.

First I went into IAM and found the Role for that particular function. I added the AWSLambdaVPCAccessExecutionRole policy to it.

Then I went into the Lambda down to the Network section, picked a VPC, then pick the subnets, and the security group I wanted.

It gave me a warning about not having outbound access, but that was ok for me because I didn't need it. My function just communicates over the VPC to my elasticsearch instance.

I have no problems re-deploying that function with up.

PS, I can still access it fine externally.

[tj]

I was reading the wrong documentation, my bad :D. I'll do some more reading, mine seems to apply fine but it just times out.

EDIT: routing in my default VPC was messed up apparently hahah