search

Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.2k forks source link

AWS Lambda cold starts #1071

Open brightwrite-admin opened 7 years ago

brightwrite-admin commented 7 years ago

I'm having issues with AWS Lambda cold starts. In despite the fact the app is high loaded, AWS Lambda will re-initialise containers from time to time anyway. Approximately once in 4 hours a few requests would take > 3 sec to return response when normal response time is 18ms. Even though it's only 0.01% of our traffic it's a major problem for us.

I'm not sure of the nature of this behaviour. But my thoughts are:

I looked quickly in handler.py. If Lambda simply forks the container it shouldn't take that long to re-init container as an app instance is cached in memory as singleton. But it happens anyway. So either AWS Lambda creates a new containers from scratch from time to time (why?) and we can't do anything about it in this case. Or if AWS Lambda doesn't persist singleton instance in memory on container forking and we can try to use a global variable in Zappa handler.py instead.

Who knows what exactly is going on here?

geeknam commented 7 years ago

It's AWS Lambda creating new containers and it's for the same reason you'd have autoscaling if you were to use EC2 instances. What was your RPM like?

wagmiwiz commented 7 years ago

FYI this is related to: https://github.com/Miserlou/Zappa/issues/851, one option is to try and keep multiple lambdas warm to minimise this happening. In our experience most of the delay is typically on AWS rather than Zappa/app side when it comes to cold starts. :/