Open psyhtest opened 1 month ago
@psyhtest Are we allowed to change the server scenario latency for the open division?
I can't see why not?
I don't recall seeing that in the rules 😇.
As usual, "anything goes" in Open.
I was wrong - this is allowed in the rules. We should get the user.conf
working.
@psyhtest You also need to place the use_token_latencies flag
mixtral-8x7b.*.use_token_latencies = 1
mixtral-8x7b.Server.ttft_latency = 3000
mixtral-8x7b.Server.tpot_latency = 300
For an Open submission, we have tried setting TTFT/TPOT latencies in
user.conf
e.g.:However, in the resulting
mlperf_log_summary.txt
we would still see:Only manually modifying
mlperf.conf
would get us the right values.This may not be specific to MoE, but to all LLM benchmarks.