apache / openwhisk

Apache OpenWhisk is an open source serverless cloud platform
https://openwhisk.apache.org/
Apache License 2.0
6.54k stars 1.17k forks source link

Invoker is not obeying action max duration (i.e. timeout) #1246

Closed rabbah closed 7 years ago

rabbah commented 8 years ago

The switch in the communication layer between invoker and action container has a 1 minute timeout that overrides the action's specified timeout --- because the connection is essentially silent up to the activation timeout. While we have a test in ActionLimitsTests for actions obeying their timeout, it does not run long enough to catch this failure.

  1. the future awaiting the container response should keep connection open up to the timeout even if "No elements passed in the last 1 minute".
  2. the test should be changed to allow for a test that runs for longer than 1 minute.

@psuter FYI.

csantanapr commented 8 years ago

@rabbah In terms of user impact, it means that any actions that customer creates with a timeout greater than 1 minute, this essentially will be ignore.

rabbah commented 7 years ago

I dont think this is valid since moving back to apache http but we need a test that holds a conection open for more than 60 seconds to assert the assumption.

loganherr commented 7 years ago

This comment is more for clarity on this issue.

This is the link I've referenced: https://github.com/openwhisk/openwhisk/blob/master/docs/reference.md#system-limits This is the command I used to create the action: wsk action create --docker -t 300000 my-action my-docker-container

The action I created sometimes takes over a minute to complete; when it takes less time it works, when it takes more it terminates early at 1 minute and crashes. Looking at the link provided, the timeout limit can be set to in the range of 100ms to 300000ms. However, if you try to set it over 60000ms (1 min) the action I created still terminates at the 1 minute marker. Is that what this issue is trying to address or should I submit another bug?

rabbah commented 7 years ago

I'll speculate you're doing a blocking invoke, in which case refer to https://github.com/openwhisk/openwhisk/blob/master/docs/actions.md#creating-and-invoking-a-simple-javascript-action section 4 (the connection is only held open for a maximum of 60s).

loganherr commented 7 years ago

It is invoked via rule and timed trigger created in the web client. Is there some way to specify this or do the rule and trigger have to be created a specific way using the CLI?

rabbah commented 7 years ago

Can you provide an example that I can try to reproduce? actions matching a rule are fire and forget and the set action limit applies.

loganherr commented 7 years ago

I create the action in the CLI using the command: wsk action create --docker -t 300000 my-action my-docker-container Afterwards I go to the web app and navigate to my action. I use the "Automate this Action" button to create a timed trigger and rule.

That's all there is to how I create it. Sometimes the action runs as intended, other times it gives a invocation timeout error and doesn't provide any logs.

rabbah commented 7 years ago

@eweiter reproduced this - will track in #1714.