Config option to return 502 on blocking activation timeout

[tysonnorris]

Since beginning to use OpenWhisk, we regularly have questions regarding blocking activations and the 60s (now configurable) timeout, where the activation submitted as "blocking" turns into "nonblocking".

I think it would be generally useful to allow the system to treat this as an error, as opposed to returning a 202 with the activation still in progress. Basic reasons are:

• Customers are confused by not having a consistent response related to the action config - e.g. setting a timeout greater than 60s (or whatever the configured blocking timeout is) would typically imply an error if duration exceeds 60s, but currently it results in 202, with activation completing "sometime later".
• We don't have any customers that actually leverage this ability, because building a client that handles 200+response as well as 202+polling is complicated.

What I suggest is:

• add a config option like downgrade-blocking-activation=true
  • if true, keep existing behavior
  • if false, return a 502 when max-wait-for-blocking-activation is exceeded, same as exceeding the action-configured timeout
• in the case of blocking timeout, indicate the action exceeded the blocking timeout, as opposed to the action timeout, e.g. The action exceeded the blocking activation time limit of 60000 milliseconds.
• due to no coordination between activation handling at controller and invoker: when timeout is past for a blocking activation, don't schedule the activation at all (since no one will ever look for the result).
• in case of "not scheduling the activation", it might be useful to let users know whether the timeout occured before processing the action, or during (e.g. was wait time too high, or init time/duration?). To do this, we can proceed with creating the activation at the invoker with a timeout error indicating "wait time too high" or "execution too long". I'm not sure creating these failure activation records is actually that useful so it might should be a separate config as well, like create-blocking-timeout-activations

[style95]

+1 !! Agree with this.

Currently, actions can be delayed up to 60s for some reason. But it might not meet the non-functional requirement in some cases. Some users may want to receive an error response in time rather than waiting for 60s.

It would be also great to add one more user-defined configuation, waitTimeout. Then users can define their hard time limit not only for execution but for initiating the activation. So if the activation processing is not initiated within this timeout, the system can respond with an error response.

[rabbah]

Re don’t schedule at all, you mean the invoker checks when a message is pulled from the queue and may drop it on the floor at that point? Or when the activation result is ready (because an activation was running already), drop the result and don’t create a record for it in the stores?

What of sequences and composition, do you envision dropping the entire sequence or having a partial result?

[tysonnorris]

RE: waitTimeout - is there a case in blocking activations where someone would care whether wait/init/duration timeout was hit, vs the single overall response timeout? This may be more interesting for non-blocking?

RE: sequences/compositions - I would expect at least the top activation to get a failure when timeout for blocking - you can end up with partially executed. I'm not sure how much timing info is currently available on the sub-activations at the invoker, to determine whether the top most has timed out, but we could add it, in which case the sub activation would get dropped at invoker. This may be a more useful case of storing failure activation before executing the action, so that a sequence can be tracked to a particular piece that took longer than expected.

[style95]

is there a case in blocking activations where someone would care whether wait/init/duration timeout was hit, vs the single overall response timeout

It would give users a more fine-grained option. For example, when a user runs a long-running action such as 30s, he can be guaranteed the invocation of his action will be initiated within 3s. He doesn't need to wait for 30s to get his action is not invoked for some reason.