Please note that detokenizing with prompt token ids is slightly slower than without. The faster the model, the bigger the influence. In my testing on internlm2-chat-1_8b, it could be about 3% slower. (49.94 RPS vs 51.55 RPS).
Update: tested internlm2-chat-7b, the influence was so small that it could be dismissed (21.984 RPS vs 21.815 RPS). The current implement performs better which could be a measurement error.
1743
Please note that detokenizing with prompt token ids is slightly slower than without. The faster the model, the bigger the influence. In my testing on internlm2-chat-1_8b, it could be about 3% slower. (49.94 RPS vs 51.55 RPS).
Update: tested internlm2-chat-7b, the influence was so small that it could be dismissed (21.984 RPS vs 21.815 RPS). The current implement performs better which could be a measurement error.