Add timeout config option when invoking lambda directly.

[t1bb4r]

Prerequisites

• [x] I am running the latest version. (up upgrade)
• [x] I searched to see if the issue already exists.
• [x] I inspected the verbose debug output with the -v, --verbose flag.
• [x] Are you an Up Pro subscriber?

Description

I'd like to call my application (lambda function) directly with cloudwatch for daily batch work which takes longer than 30 seconds. I've increase the timeout on my lambda function, but there is still a timeout in up which I can't modify. I did find a similar request https://github.com/apex/up/issues/751. I'm asking for the something similar with another reason. Please give us a way to increase the timeout when calling lambda directly and not through api gateway.

I understand as a workaround I can deploy "background work" as a separated function and call it, but up is already configured with all my CI pipelines and environment variables and I really like using up! This would be an amazing feature for me.

Steps to Reproduce

Deploy an application which takes longer than 15 seconds to complete:

const Koa = require("koa");
const app = module.exports = new Koa();

app.use(async function(ctx) {
  await new Promise((resolve) => setTimeout(resolve, 60000));
  console.log("Blocking log");
  ctx.body = "Hello World";
});

if (!module.parent) app.listen(process.env.PORT || 3000);

Run the following commands:

up
aws lambda update-function-configuration --function-name apex-timeout --timeout 100 --region eu-west-1
aws lambda invoke --function-name apex-timeout --cli-binary-format raw-in-base64-out --payload '{ "path": "/" }' out --log-type Tail --region eu-west-1 --query 'LogResult' --output text |  base64 -d

Output:

.
.
.
2020/08/31 11:21:43 http: proxy error: net/http: timeout awaiting response headers
{"fields":{"app":"apex-timeout","duration":15021,"id":"","ip":"","method":"GET","path":"/","plugin":"logs","query":"","region":"eu-west-1","size":0,"stage":"staging","status":502,"version":"$LATEST"},"level":"error","timestamp":"2020-08-31T11:21:43.092673704Z","message":"response"}
END RequestId: ***
REPORT RequestId: ***   Duration: 16129.38 ms   Billed Duration: 16200 ms   Memory Size: 512 MB Max Memory Used: 98 MB  Init Duration: 182.57 ms    

When I try to increase the timeout to 100 using .proxy.timeout:

Error: initializing: reading config: validating: .proxy: .timeout: should be <= 25

up.json for reference:

{
  "name": "apex-timeout",
  "regions": [
    "eu-west-1"
  ]
}

Love Up?

Please consider signing up for Up Pro (https://up.docs.apex.sh/#guides.subscribing_to_up_pro) or donating via https://opencollective.com/apex-up/donate.

Slack

Join us on Slack https://chat.apex.sh/

EDIT: updated aws cli command to work with cli version 2

[tj]

Sounds pretty reasonable to me! The only problem I guess is that you can only really have one timeout configuration, so if you set it above ~30 it will lead to API Gateway handling the timeout instead of Up. I think the response was always plain-text or maybe it was JSON I forget, but there's not much control over that response

[t1bb4r]

That is an unfortunate drawback.

Could the timeout be altered through the payload? If we can't do it directly, maybe some kind of reserved header.

aws lambda invoke --function-name apex-timeout --payload '{ "upTimeout": 100 }'
aws lambda invoke --function-name apex-timeout --payload '{ "headers": { "UP-TIMEOUT": 100} }'

When invoking directly we won't get the API Gateway timeout while the application will still keep control over normal timeout responses.

[tj]

Hmm yeah you're right, a header field should work fine, I'll check that out this week

[t1bb4r]

Hi @tj. This works really well in 1.6.2, thank you so much!

I still have one small nuisance, the lambda timeout gets changed each time I deploy so I need to do up && aws lambda update-function-configuration --function-name apex-timeout --timeout 900, which I can do in a postdeploy hook, but still need the aws cli installed and configured on every system.

Could we go back to having 2 timeouts, one for lambda, one for proxy? From my understanding the proxy will timeout before API Gateway so the user experience will be the same, and if we choose to increase the lambda timeout for background work the lambda timeout is on us.

[tj]

Hmm if you set the proxy.timeout to 900 and X-Up-Timeout that should be enough right? That should give you the override for background work, and the default for the API Gateway UX

[t1bb4r]

I see... I can set timeout to 25 seconds on "normal" requests through the header and leave proxy.timeout 900. That would work, but there is still a validation of maximum 25 seconds on the proxy.timeout

$ up version
1.6.2-pro
$ up
Error: initializing: reading config: validating: .proxy: .timeout: should be <= 25

[tj]

Oh sorry what I said doesn't make sense actually, not sure what I was thinking haha. The regular proxy.timeout should be ok for the 25 seconds for API Gateway, and then this X-Up-Timeout for internal invocations? I thought that was the intention of d5e15dfe

[t1bb4r]

Yes, that is the way I wanted to use it, but up is overwriting my increased lambda timeout on each deploy. Which is why I suggested we use two timeouts in the up config, one for proxy and one for lambda.

[tj]

You're right sorry, my brain wasn't working yesterday haha. I'll add an option and maybe just default it to something higher

[tj]

up upgrade should give you v1.7.0-pro with lambda.timeout defaulting to 60 seconds

[t1bb4r]

Just tested it and it works perfectly. Thanks!

[tj]

awesome!

[Dankimhaejun]

@tj @t1bb4r I read your comment, and I tried to change timeout but it is not working... how can I solve it..?

[t1bb4r]

Hi @Dankimhaejun, the fields you are looking for is lambda.timeout and proxy.timeout. proxy.timeout is still capped at 25, because of the api gateway restriction, see the docs for more information.

  "lambda": {
    "timeout": 900
  },
  "proxy": {
    "timeout": 25
  },

[tj]

Yeah unfortunately HTTP requests from API Gateway can't be more than 30s at the moment, this other timeout only works if you invoke Lambda directly

[XavM]

Hi there,

Sorry to dig this up, and i know that #814 is still not documented, but I could need some help with the "undocumented documentation" ;)

When setting the up.json lambda.timeout and invoking Lambda functions directly (no API Gateway), I still get some "net/http: timeout awaiting response headers" before the lambda.timeout

up version: 1.7.0

Ex: up.json

  "proxy": {
    "timeout": 25,
    "listen_timeout": 25
  },
  "lambda": {
    "memory": 128,
    "timeout": 120
  },

(To make things a bit more fuzzy, that timeout always comes long before the lambda.timeout, but sometimes after the proxy.timeout)

Adding the X-Up-Timeout header when invoking the Function seams to fix that issue

Ex: lambda invoke that always run until the lambda.timeout

aws lambda invoke --function-name 'my-fun' --cli-binary-format raw-in-base64-out   --payload '{
    "Headers": { "X-Up-Timeout": "120" },
    "path": "/"
  }' out --log-type Tail --region eu-west-3 --query 'LogResult' --output text |  base64 -d

The question : Do we need to set up the lambda.timeout and invoke the function with the X-Up-Timeout header to allow that function to run longer than proxy.timeout ?

If "yes" : Which lambda timeout setting will prevail ? (assuming you feel like setting 2 distinct values ...) if "no" : Should the lambda.timeout be enough, and why do I get those timeouts long before the lambda.timeout ?

---

BTW : Apex Up is a real joy to work with, just like the many other things you have done so far; Thx for all of that great job @tj !

[tj]

@XavM good question :D, you would still have to set both in the direct-invocation scenario. The lambda.timeout is for the Lambda function's timeout, and proxy.timeout is for Up's proxy server which sends the request to your app, but even when you're bypassing API Gateway the proxy server is still used, but you no longer have the API Gateway 30s timeout limitation.

It's a bit awkward, but Up wasn't really designed to be a FaaS replacement, so I wouldn't rely on this too much personally.

[XavM]

Crystal clear, and thank you for the quick reply !!