search

flexflow / FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving
https://flexflow.readthedocs.io
Apache License 2.0
1.59k stars 218 forks source link

Define several request manager states and added new APIs #1370

Closed zikun-li closed 1 month ago

zikun-li commented 2 months ago

Description of changes:

  1. RequestManager::Status is modified to indicate the serving status. Previous status is renamed to RequestManager::BackgroundServerStatus.
  2. Two APIs are added.

This change is Reviewable

jiazhihao commented 1 month ago

@zikun-li What's the status of this PR?

zwang86 commented 1 month ago

@jiazhihao Most changes in this branch has been applied to the specscheduler branch. There is some latest changes in specscheduler branch we may need to keep up with.