Open connolly opened 4 years ago
(note: copying my comments from the Slack thread)
It's tough to speed it up w/o preallocating a few machines (i.e., set up the autoscaler to always keep a buffer of ~N free machines, to be immediately available when the next user(s) connect). But that costs money.
A workaround is to warn the user that the cluster will take 10 minutes to spin up. They'll be less annoyed if they're aware of this (and will incorporate the delay into the workflow).
One thing to look into: Fargate -- https://aws.amazon.com/fargate/ -- this is supposed to enable one to run containers w/o specifying a server on which to run them. I'm not sure if that means the spinup is faster. The thing to look at is how they intract with EKS -- amazon just announced the tie-in on reInvent, but I didn't get a chance to read about it. It does look potentially promising:
Current deployment from login to spark cluster is about 10 mins (5 mins to deploy the EC2 VMs and 5 mins to deploy the pod). Break down of timing from @stevenstetzler is