harris-chris
commented
1 year ago Yes, this is quite confusing. The idea with Jot, which seemed like a good idea when I wrote it, is to try to spread the computation work intelligently between the three steps below:
- Image build time: pack as much of the work as possible into this stage
(which only runs once, and entirely outside of AWS). This includes
instantiating and (if
package_compileis requested) compiling. Within the Dockerfile, aninit.jlscript gets run that does this work. - Function instantiation time: when the lambda function is instantiated, the HTTP server that handles incoming function calls starts. The precompiled/cached data that Julia needs should already exist in the DEPOT_PATH, so this should be quick. 3: Function call time: when a function call arrives, the actual responder function is called. As with the HTTP server this should be cached/precompiled, so again this should be quick.
When testing against a local docker image (entirely outside of AWS),
function response times are much faster with package_compile set to true,
so (outside of AWS) the logic above seems to be working.
It is hard to infer why AWS is taking so long in your situation. If provisioned concurrency is on, then only stage 3 of the stages above is being run, so it should be very quick.
Rather that trying to guess what's going on, I think it's probably easiest to add some good debug output to the code that gets called in stages 1-3 above, and then to add an option to run the function in this debug mode. I think I can do that in the next couple of days.
If I get some time today I'll check that times I'm getting running the
image locally vs via Lambda, and with package_compile on vs off.
How often do you see that 60-second response time? Is it literally just the first time you run a newly-created function?
On Tue, 22 Nov 2022, 01:30 Matt Doak, @.***> wrote:
We're seeing initial execution times of 60s, while subsequent calls are 20ms. While we had thought this was a Lambda cold start issue, we've set up provisioned concurrency and confirmed with support that provisioned instances are being used:
"...you can see that the latency is equally long when it is provisioned concurrent invocation. Therefore ... the provisioned concurrency is correctly set.
They seem to suggest that there are two separate sections: Lambda handler and core logic.
"We recommend that you separate the code inside the lambda handler so that these long-time logic can be executed in the init phase."
Here's the document they referenced:
https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code
I still don't understand why we would be seeing subsequent requests coming in at 20ms. Seems like something is being cached and reused. I would assume that having provisioned instances would already have the initialization code ready, but maybe that switch to core logic is required for that step. Any thoughts?
Some more context:
Again, the initial phase of the function does not exist during provisioned concurrency because the initial phase is set in advance. However, if the logic that takes a long time does not run during init duration, it cannot be affected by provisioned concurrency setting.
To sum up, you need to save time by separating the time-consuming initialization logic outside the Lambda handler.
— Reply to this email directly, view it on GitHub https://github.com/harris-chris/Jot.jl/issues/37, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALDMVSHHHHAKNLCRHPR4MWLWJOPRFANCNFSM6AAAAAASG2QPMY . You are receiving this because you are subscribed to this thread.Message ID: @.***>
sklink
We're seeing initial execution times of 60s, while subsequent calls are 20ms. While we had thought this was a Lambda cold start issue, we've set up provisioned concurrency and confirmed with support that provisioned instances are being used:
They seem to suggest that there are two separate sections: Lambda handler and core logic.
Here's the document they referenced: https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code
I still don't understand why we would be seeing subsequent requests coming in at 20ms. Seems like something is being cached and reused. I would assume that having provisioned instances would already have the initialization code ready, but maybe that switch to core logic is required for that step. Any thoughts?
Some more context: