Support for more feature rich schedulers - Githubissues

definitive-io / LLProxy

Large Language Proxy

Apache License 2.0

6 stars 1 forks source link

Support for more feature rich schedulers #4

Open cwelton opened 1 year ago

cwelton commented 1 year ago

Current scheduler is purely FIFO.

Desirable enhancements:

gradiated scheduling to slow things down more gently as capacity lowers
priority queuing
etc