This pull request introduces a benchmarking script for evaluating the performance of LLama's generation process. The script, located at bin/benchmark/main.rs, aims to measure the tokens/second average and standard deviation across multiple runs. The benchmarking is achieved by executing the following command:
cargo run --bin sample <model_name> <tokenizer_filepath> <prompt> <n_tokens> <repetitions>
Changes Made
Added the benchmarking script at bin/benchmark/main.rs.
Overview
This pull request introduces a benchmarking script for evaluating the performance of LLama's generation process. The script, located at
bin/benchmark/main.rs, aims to measure the tokens/second average and standard deviation across multiple runs. The benchmarking is achieved by executing the following command:Changes Made
bin/benchmark/main.rs.README.mdwith benchmark