hbbr0312
commented
5 days ago The current simulator runs only one iteration and then exits. If you want to experiment with multiple iterations and generate multiple output tokens, you'll need to modify the code in Scheduler.cc.
To explain the code flow: the simulator creates a model program for the given requests (which acts as a kind of compiler), measures the cycles through simulation, and then exits. To run multiple iterations, you'll need to generate and execute a model program for each iteration.
Hello! I find that in npu-only mode, the output size is set to be 1 but I want to use this simulator to generate tokens more than 1. I use stats.tsv in share-gpt folder as cli_config( it contains two columns: input size and output size while request-traces/rr/share-gpt2-bs512-ms7B-tp4-rr-0.csv has seq_len and ch_idx) and set the output size to be cli_config.row[x].second but an error occurs! I want to ask you how to use this simulator to generator multiple output tokens(or use stats.csv in share-gpt2 folder),Thanks!