Vidur is a high-fidelity and extensible LLM inference simulator. It can help you with:
... all without access to GPUs except for a quick initial profiling phase.
Please refer to our MLSys'24 paper for more details. We have a live demo that captures the capabilities of the system.
Model / Device | A100 80GB DGX | H100 DGX | 4xA100 80GB Pairwise NVLink Node | 8xA40 Pairwise NVLink Node |
---|---|---|---|---|
meta-llama/Meta-Llama-3-8B |
✅ | ❌ | ✅ | ❌ |
meta-llama/Meta-Llama-3-70B |
✅ | ❌ | ✅ | ❌ |
meta-llama/Llama-2-7b-hf |
✅ | ✅ | ✅ | ✅ |
codellama/CodeLlama-34b-Instruct-hf" |
✅ | ✅ | ✅ | ✅ |
meta-llama/Llama-2-70b-hf |
✅ | ✅ | ✅ | ✅ |
internlm/internlm-20b |
✅ | ✅ | ✅ | ✅ |
Qwen/Qwen-72B |
✅ | ✅ | ✅ | ✅ |
Llama3-8B
and Llama3-70B
which support 16k context length by passing additional CLI params:For random forrest:
--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size 16384 \
--random_forrest_execution_time_predictor_config_prediction_max_batch_size 512 \
--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request 16384 \
For linear regression:
--linear_regression_execution_time_predictor_config_prediction_max_prefill_chunk_size 16384 \
--linear_regression_execution_time_predictor_config_prediction_max_batch_size 512 \
--linear_regression_execution_time_predictor_config_prediction_max_tokens_per_request 16384 \
Vidur exports chrome traces of each simulation. The trace can be found in the simulator_output
directory. The trace can be opened by navigating to chrome://tracing/
or edge://tracing/
and loading the trace.
mamba
To run the simulator, create a mamba environment with the given dependency file.
mamba env create -p ./env -f ./environment.yml
mamba env update -f environment-dev.yml
venv
cd
into the repository rootvenv
module using python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
deactivate
to deactivate the virtual environmentconda
(Least recommended)To run the simulator, create a conda environment with the given dependency file.
conda env create -p ./env -f ./environment.yml
conda env update -f environment-dev.yml
First, setup your account on https://<your-org>.wandb.io/
or public wandb, obtain the api key and then run the following command,
wandb login --host https://<your-org>.wandb.io
To opt out of wandb, pick any one of the following methods:
export WANDB_MODE=disabled
in your shell or add this in ~/.zshrc
or ~/.bashrc
. Remember to reload using source ~/.zshrc
.wandb_project
and wandb_group
as ""
in vidur/config/default.yml
. Also, remove these CLI params from the shell command with which the simulator is invoked.To run the simulator, execute the following command from the repository root,
python -m vidur.main
or a big example with all the parameters,
python -m vidur.main \
--replica_config_device a100 \
--replica_config_model_name meta-llama/Llama-2-7b-hf \
--cluster_config_num_replicas 1 \
--replica_config_tensor_parallel_size 1 \
--replica_config_num_pipeline_stages 1 \
--request_generator_config_type synthetic \
--length_generator_config_type trace \
--interval_generator_config_type static \
--[trace|zipf|uniform|fixed]_request_length_generator_config_max_tokens 4096 \
--trace_request_length_generator_config_trace_file ./data/processed_traces/arxiv_summarization_stats_llama2_tokenizer_filtered_v2.csv \
--synthetic_request_generator_config_num_requests 128 \
--replica_scheduler_config_type vllm \
--[vllm|lightllm|orca|faster_transformer|sarathi]_scheduler_config_batch_size_cap 256 \
--[vllm|lightllm]_scheduler_config_max_tokens_in_batch 4096
The simulator supports a plethora of parameters for the simulation description which can be found here.
The metrics will be logged to wandb directly and a copy will be stored in the simulator_output
directory along with the chrome trace. A description of all the logged metrics can be found here.
To format code, execute the following command:
make format
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.