[Jobs] Managed job controller process taking too much memory during peak time

[Michaelvll]

We only limit the number of parallel running managed jobs with the number of CPU core each job controller process uses, but that is not enough because:

1. during peak time, when there are multiple parallel jobs doing sky launch it can still experience OOM.
2. when a user is using a controller instance with memory/cpu ratio <4.

A potential solution:

1. Adopt similar resource management from SkyServe to limit the number of parallel launches to guard against high memory consumption.

To reproduce:

for i in `seq 1 100`; do
  sky jobs launch --fast -n test-job-$i -yd --fast "echo hi; sleep 240" &
done   
wait

If we run sky jobs queue after a while, we can see FAILED_CONTROLLER for the some of the jobs (mainly because the OOM)

Version & Commit info:

• sky -v: PLEASE_FILL_IN
• sky -c: PLEASE_FILL_IN

[cblmemo]

I'm following SkyServe's implementation to apply CPU limits to sky.launch. At the same time also memory limits to number of concurrent jobs for now. Do we have some results showing the OOM is caused by sky.launch but not ray job management? cuz I remember otherwise but feel free to correct me if I'm wrong ;)