FastAPI server that handles provision requests (sky launch) in the background

[concretevitamin]

A user has the following setup:

• a FastAPI/gunicorn server that handles provision requests
• the handler calls sky launch (detach setup + detach run) to provision a cluster to do some work
• however, this is a blocking call, and it freezes up a worker in the web server for ~5-10 minutes until the cluster is up
• a few more requests later, this makes the server quickly unresponsive to new requests

We should investigate this usage pattern and add a demo example, perhaps by using “background tasks”: https://fastapi.tiangolo.com/tutorial/background-tasks/

[concretevitamin]

Seems to work well (with detach_setup and detach_run to avoid #2182):

"""Example: a FastAPI server that serves by calling `sky launch`.

Usage:

    $ python server.py

    # In another terminal:
    $ curl localhost:8000
    # Or:
    $ for i in $(seq 32); do curl http://0.0.0.0:8000; done
"""
import uuid

import fastapi

import sky

app = fastapi.FastAPI()

def background_task(task_id: int):
    task = sky.Task(run='echo hi').set_resources(
        sky.Resources(cpus=2, use_spot=True))
    sky.launch(
        task,
        cluster_name=f'sky-{task_id}',
        down=True,
        idle_minutes_to_autostop=1,
        # Use these to avoid tailing logs:
        detach_setup=True,
        detach_run=True,
    )

@app.get('/')
async def root(background_tasks: fastapi.BackgroundTasks):
    task_id = str(uuid.uuid4())[-6:]
    background_tasks.add_task(background_task, task_id)
    return {'message': f'Background task `sky launch` started: {task_id}'}

def main():
    import uvicorn
    uvicorn.run(app, host='0.0.0.0', port=8000)

if __name__ == '__main__':
    main()

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[Michaelvll]

This is being added in #2735

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.