Lightning-AI / LitServe

High-throughput AI model serving engine with a friendly interface and breathtaking scale.
https://lightning.ai/docs/litserve
Apache License 2.0
128 stars 13 forks source link

Feat: Add support for fastapi lifespan events #110

Open robTheBuildr opened 1 month ago

robTheBuildr commented 1 month ago

🚀 Feature

Add the ability for developers to customize the FastAPI lifespan, startup, and 'shutdown` event handlers.

Motivation

Users can use this to setup logging or resource management.

Pitch

There are 2 options here.

1) Add a function attach_event_handler(event: str, func: Callable) which takes either 'lifespan', 'startup' or 'shutdown' as the event and applies the given logic.

2) More explicit and abstracting away FastAPI Lifespan event knowledge from the user, add during_lifespan(func: Callable), on_startup(func: Callable), and on_shutdown(func: Callable) that the user can call on Server to apply these lifespan handlers.

Alternatives

Add custom template startup and shutdown event handler for logging and resource management. This will get cumbersome.

Additional context

github-actions[bot] commented 1 month ago

Hi! thanks for your contribution!, great first issue!

aniketmaurya commented 1 month ago

hi @robTheBuildr, we are using the FastAPI lifespan to manage our parallel inference workers. May I know more about your use case and what you are trying to achieve with lifespan?

williamFalcon commented 1 month ago

@robTheBuildr what will be helpful is a reproducible code snippet we can run that shows something you are trying to do, but not able to do with litserve