Open Weixuanf opened 1 month ago
It might take several minutes to ten minutes to cold start, here are some steps:
Actually all GPU instance types (like g4dn/g5/g6) have instance store, boot time is the same whether you use instance store or not.
The solution uses instance store to improve models loading and switching (inside ComfyUI) performance.
It's a tradeoff, we spend more time setting up the environment to get better performance.
thanks very much for your reply. I want to run serverless comfyui servers that scale down to 0 when no requests to save time. So cold start time is very important for me. I hope to get < 5s cold start time (excluding comfyui boot time itself). I'm thinking EC2 + EBS and stopping/starting the EC2 server to achieve better cold start times than using auto scale group. If you have other suggestions, please let me know!
Actually all GPU instance types (like g4dn/g5/g6) have instance store, boot time is the same whether you use instance store or not.
oh so even if I use EKS, there will still be instance store in it?
Yes, g4dn & g5 & g6 all have instance store, refer to Amazon EC2 instance store, you can use it or just ignore it (it's free).
EC2 with EBS will have less boot time, because there's no image pulling and model syncing. But you need to handle EC2 scaling in/out yourself. Besides that, loading models from EBS to GPU memory might take more time than loading from instance store.
thanks very much for your reply. I want to run serverless comfyui servers that scale down to 0 when no requests to save time. So cold start time is very important for me. I hope to get < 5s cold start time (excluding comfyui boot time itself). I'm thinking EC2 + EBS and stopping/starting the EC2 server to achieve better cold start times than using auto scale group. If you have other suggestions, please let me know!
Actually all GPU instance types (like g4dn/g5/g6) have instance store, boot time is the same whether you use instance store or not.
oh so even if I use EKS, there will still be instance store in it?
Have you thought about mounting EFS to your instances?
Cuz I want to scale down to 0 instance when there’s no request.
How long does it take to cold start EC2 from 0 instance? I think instance store EC2 is slower to boot than EBS backed EC2 instance?
and downloading models from S3 to instance store takes extra time. How’s the downloading speed look like from s3 to instance store?
Thanks for this amazing template!