Closed Scott-Hoefer-Spycloud closed 2 months ago
Hey @Scott-Hoefer-Spycloud, thanks for reaching out 🙏
It looks like the function you're trying to power-tune is returning an error when invoked. And it looks like it's returning that long list of number as invocation response.
Basically, Lambda Power Tuning assumes that your function runs without raising errors. In case of errors, it assumes that something went wrong and it stops power-tuning.
A few questions:
@alexcasalboni Thanks for the response
Uh, that is interesting 🤔
The invocation results aren't really manipulated though. Since you aren't using any pre/post-processor, the list of numbers is simply invocationResults.Payload
. And it's an error because it found a truthy value for invocationResults.FunctionError
.
A few more questions:
{
"errorType": "Error",
"errorMessage": "Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,34,50,48,50,52,45,48,52,45,48,50,84,49,52,58,52,50,58,53,48,46,48,54,53,90,32,51,55,51,101,56,48,101,50,45,49,51,97,54,45,52,49,52,48,45,98,100,98,102,45,101,50,57,52,50,54,55,49,100,50,50,50,32,84,97,115,107,32,116,105,109,101,100,32,111,117,116,32,97,102,116,101,114,32,50,57,46,48,50,32,115,101,99,111,110,100,115,34,125 with payload \"...",
"trace": [
"Error: Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,34,50,48,50,52,45,48,52,45,48,50,84,49,52,58,52,50,58,53,48,46,48,54,53,90,32,51,55,51,101,56,48,101,50,45,49,51,97,54,45,52,49,52,48,45,98,100,98,102,45,101,50,57,52,50,54,55,49,100,50,50,50,32,84,97,115,107,32,116,105,109,101,100,32,111,117,116,32,97,102,116,101,114,32,50,57,46,48,50,32,115,101,99,111,110,100,115,34,125 with payload \"...",
" at runInSeries (/var/task/executor.js:167:19)",
" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
" at async module.exports.handler (/var/task/executor.js:65:19)"
]
}
Again, I can use the same payload and get a successful response outside the tool so I know thats correct
@alexcasalboni So I switched from a multiple weighted payload to a single payload input and it works now, no other change to the payload was made. Not sure what the issue is with weighted payloads but I can use it with just one at a time for now. Feel free to close this if you want or ask further questions. Thanks!
Thanks, that makes a lot of sense now 🚀
It definitely looks like an issue and we've recently changed how weighted payloads are treated recently so it could be a regression.
Could you please share with us the structure of the weighted payload you were using? (obfuscating the actual payload, if needed)
FYI @andrestoll
{
"input": {
"lambdaARN": "<my lambda arn>",
"powerValues": [
2048,
4096,
5120,
6144,
8192,
10240
],
"num": 50,
"strategy": "balanced",
"payload": [
{
"weight": 30,
"payload": {...}
},
{
"weight": 20,
"payload": {...}
}
]
}
Can you check the logs of the "executor" function if the event payload for the invocation is what you expect? Is there a difference when using a weighted payload and single payload? Maybe you can share as well
Btw it seems you are passing in \"{}\"
as the individual payload to the weighted payload is that what you need?
@andrestoll Payload isn't really empty when I run it I just can't post the contents here. I can check the executor logs and get back to you
@Scott-Hoefer-Spycloud I cannot reproduce the error, if you can let me know what caused the function to be tuned to fail we might have some more ideas.
@alexcasalboni invocationResults.Payload
is a uint8 array, hence the list of numbers I suggest we include out the invocationResults.FunctionError
instead
@andrestoll I am not sure what is causing the error as the error message is less than helpful and it works when I invoke the lambda manually with the same input. It does seem to be tied to one input whereas others work, but again I am unable to tell what the issue is without a meaningful error message. I did confirm that the weighted payload works if I remove the aforementioned input however
@Scott-Hoefer-Spycloud If you go to the log group /aws/lambda/[stack-name]-executor-[suffix]
you will find more information on the invocation that caused the error and more important the payload of it.
Look for "INFO Invoking function"
also your Lambda function logs should give you some information why your function failed, can you confirm that the payload is exactly the same?
I was having the same issue, and for me the problem turned out to be that my Lambda function was timing out. I actually went in and modified the executor's code to dump the FunctionError
property as well to the log, which gave me the value Unhandled
- which then got me to triple check the logs of my Lambda function, and realize that it was timing out. Not sure if we can improve the error message in the power tuning tool to catch this scenario - or at least give a hint? :)
Thank you @lachriz-aws this sounds like a great step forward!
I'll make sure to investigate if that's reproducible with any function that times out 👌
@Scott-Hoefer-Spycloud @lachriz-aws we have a working implementation here (merging soon): #247
It will show the proper error message instead of the weird array buffer :)
The fix has been merged 🎉
Closing this for now, let me know if you still encounter the problem.
Running into a weird error message that I can't decipher when trying to run the state machine, happens with any payload. Seems to originate from the utils.invokeLambdaWithProcessors function