James-QiuHaoran / LLM-serving-with-proxy-models

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
Apache License 2.0
16 stars 5 forks source link

Add support for training per-LLM predictor #5

Closed James-QiuHaoran closed 3 months ago

James-QiuHaoran commented 3 months ago

Support for training a customized predictor for a specific LLM model by adding a flag that specifies the model name from the dataset

James-QiuHaoran commented 3 months ago

Added in commits:

James-QiuHaoran commented 3 months ago

Example commands:

# predictor training for a particular LLM model
python preprocess_dataset.py --task_type 0 --data_size 100 --model_name "gpt-4"
python latency_prediction.py --task_type 0 --data_size 100 --model_name "gpt-4"

Note that the model names are from LMSYS-Chat-1M dataset.