Plugin timing out without doing anything when running in VPC

[juanjoDiaz]

Hi,

I'm currently using Serverless 1.13.2.and using serverless-plugin-warmup to warmup my lambdas.

The warmup lambda is correctly invoked every 5 minutes. However, it doesn't actually call any of my lambdas and eventually times out. I've increased the time out to several minutes and it still doesn't call anything and simply times out.

The logs don't tell much either:

START RequestId: 37f7c285-4b45-11e7-85cb-c75ba9915e76 Version: $LATEST
2017-06-07T05:50:33.990Z    37f7c285-4b45-11e7-85cb-c75ba9915e76    Warm Up Start
END RequestId: 37f7c285-4b45-11e7-85cb-c75ba9915e76
REPORT RequestId: 37f7c285-4b45-11e7-85cb-c75ba9915e76  Duration: 10000.97 ms   Billed Duration: 10000 ms Memory Size: 128 MB   Max Memory Used: 32 MB
2017-06-07T05:50:43.982Z 37f7c285-4b45-11e7-85cb-c75ba9915e76 Task timed out after 10.00 seconds

Any hint of what the problem might be?

[goncaloneves]

Could you place the warm up lambda code here and check the logs of the service lambdas to confirm they are not being invoked. What is your service runtime?

[juanjoDiaz]

I checked the logs. The lambdas are not being called by the warmer. My runtime is node 6.10.2 The generated lambda looks like:

"use strict";

/** Generated by Serverless WarmUP Plugin at 2017-06-07T11:37:46.127Z */
const aws = require("aws-sdk");
aws.config.region = "eu-central-1";
const lambda = new aws.Lambda();
const functionNames = "MyLambda1,MyLambda2,MyLambda3,MyLambda4,MyLambda5".split(",");
module.exports.warmUp = (event, context, callback) => {
  let invokes = [];
  let errors = 0;
  console.log("Warm Up Start");
  functionNames.forEach((functionName) => {
    const params = {
      FunctionName: functionName,
      InvocationType: "RequestResponse",
      LogType: "None",
      Qualifier: process.env.SERVERLESS_ALIAS || "$LATEST",
      Payload: "{\"source\": \"serverless-plugin-warmup\"}"
    };
    invokes.push(lambda.invoke(params).promise().then((data) => {
      console.log("Warm Up Invoke Success: " + functionName + "", data);
    }, (error) => {
      errors++;
      console.log("Warm Up Invoke Error: " + functionName + "", error);
    }));
  });
  Promise.all(invokes).then(() => {
    console.log("Warm Up Finished with " + errors + " invoke errors");
    callback();
  }).catch(console.log);
}

After logs of debugging it seems like:

lambda.invoke(params).promise().then((data) => {
      console.log("Warm Up Invoke Success: " + functionName + "", data);
    }, (error) => {
      errors++;
      console.log("Warm Up Invoke Error: " + functionName + "", error);
    })

never returns...

[goncaloneves]

The only reason I can think of is the invokes are still pending with superior timeouts, so basically all invokes are still waiting for the callback call.

What are you're doing on your invoked lambdas? Are you checking for the event.source and calling callback to the caller (warmup)?

[juanjoDiaz]

Yeah. I check for the event.sourceand return immediately in all lambdas. Tried max timeout and memory limit but still same issue.

If I run the function locally it seems to work just fine. But not on lambda.

[goncaloneves]

You mean the warm up function?

I am finding hard to understand where is breaking. What you could do is some try tests by rewriting the live warm up lambda until you find the issue. Let me know if you will do that and get to the cause. Share your findings.

The same way if anyone is having this issue please share some knowledge.

I will keep the issue open in the meantime.

[juanjoDiaz]

Alright! The issue was that lambdas deployed to a VPC should be deployed to a private subnetwork with a NAT gateway in order to access other AWS resources or the public internet.

I've added that information to the Readme in the PR #14 . You can close the issue once you merge it.

Thanks for the help!

[goncaloneves]

Thanks @juanjoDiaz ! 👌

Strange why it keeps in pending state, ideally should have thrown. Well... tough one 😉

Thanks for solving this and your half day of work for the community hehe

[contrarianaupong]

You can set the warmup to use no VPC, while others functions with VPC Simply override the vpc setting in warmup by adding vpc: securityGroupIds: [] subnetIds: []

Here is the reference. https://github.com/serverless/serverless/issues/4387#issuecomment-337116499

[JerseyGood]

@contrarianaupong 's way works

but need to hack into addWarmUpFunctionToService() (I don't other ways around)

   /** SLS warm up function */
    this.serverless.service.functions.warmUpPlugin = {
      description: 'Serverless WarmUP Plugin',
      events: this.warmup.schedule.map(schedule => ({ schedule })),
      handler: this.pathHandler,
      memorySize: this.warmup.memorySize,
      name: this.warmup.name,
      runtime: 'nodejs8.10',
      package: {
        individually: true,
        exclude: ['**'],
        include: [this.folderName + '/**']
      },
      timeout: this.warmup.timeout,
      // override provider level of VPC settings.
      vpc: {
        securityGroupIds: [],
        subnetIds: []
      }
    }

A better alternative would be Not using Provider VPC settings and set Function VPC Configurations one by one, which leaves warmup function "out of VPC"