Many "Message length exceeded" from a subject will result in lock out (for that subject)

[jlost]

Environment details:

• Ubuntu 14.04.5 LTS
• Docker version 1.12.0, build 8eab29e
• ansible 2.1.2.0

Steps to reproduce the issue:

1. Invoke an Action which returns a message that exceeds the 1MB limit. Observe that even if you invoke it as "blocking" and the invocation is short, you will wait until timeout and receive only the Activation ID in the response. Get the Activation, observe that the duration is short in spite of the long wait.
2. Repeat step 1 (invoke as non-blocking or you'll be waiting a very long time) until you get the error "Too many requests for namespace". Check those Activations and verify that they have completed with the "message length exceeded maximum" error.
3. Wait a few minutes (or any arbitrary length of time) and try an Action invocation. Observe that you still get the "Too many requests" error, even though none of the requests you made in step 2 are still in flight.

Provide the expected results and outputs:

The result I would expect is that in step 1, a "message length exceeded" error is returned to the caller, and in step 3, "Too many requests" would either not occur (because they're quickly being completed, not still in-flight), or would go away after a short time.

Provide the actual results and outputs:

Blocking invocations only return the Activation ID after a timeout, and the OW instance becomes locked with "Too many requests". This seems like a Denial of Service vulnerability as well.

Code Samples

break-openwhisk.js:

exports.main = (args) => {
    let bigstring = "";
    for (let i = 0; i < 5000000; i++) {
        bigstring += "A";
    }
    return { bigstring: bigstring, message: "Should have failed" };
};

spam.sh:

#!/bin/bash

for i in {1..200}
do
    echo "invocation #$i..."
    wsk action invoke -iv break-openwhisk
    sleep 1
done

[rabbah]

You are right - this will result in a subject being throttled until the accounting is reset. This is a related proposed patch: https://github.com/rabbah/openwhisk/commit/c8a31abf51b5b00eceb6d82a833a3e96fc642c6b (although does not address this particular issue yet which requires a bit more refactoring).

[rabbah]

This issue is a known defect and related to https://github.com/apache/incubator-openwhisk/issues/2118#issuecomment-292360083.

[jlost]

Thanks for the info! Is there any reason that patch hasn't been accepted into the codebase yet?

[rabbah]

I haven't opened the pull request yet (It's part of several inter-related fixes). I expect to do that very soon.