search

microsoft / vidur

A large-scale simulation framework for LLM inference
MIT License
255 stars 36 forks source link

Supported Simulation Parameters Link Doesn't Work #36

Open ozcanmiraay opened 4 weeks ago

ozcanmiraay commented 4 weeks ago

Hello! There is a statement in the README file: "The simulator supports a plethora of parameters for the simulation description which can be found here."

However, the link doesn't work: https://github.com/microsoft/vidur/blob/main/docs/launch_parameters.md

How can we solve this? Thanks!

AgrawalAmey commented 4 weeks ago

@ozcanmiraay, we recently removed the docs page. All the params are now self documented, you can check them by running with -h flag.

ozcanmiraay commented 3 weeks ago

@AgrawalAmey Thank you so much! Also, how can I see the supported bounds for each of the numeric parameters? E.g., is there an upper/lower bound for a parameter like scheduler_config_batch_size_cap or request_length_generator_config_max_tokens enforced by how the simulation was set up?

nitinkedia7 commented 6 days ago

@ozcanmiraay, for models other than LLama3 ones, scheduler_config_batch_size_cap = 128 and request_length_generator_config_max_tokens = 4096 are the maximum. For llama3, the maximums are 512 and 16k respectively. Some more details regarding this are available at https://github.com/microsoft/vidur?tab=readme-ov-file#supported-models .