[Langchain-Chatchat]Add time consumption msg about first token and rest tokens

[johnysh]

The current model is unable to calculate the time spent on first token and rest tokens, can we add this msg ?

[Oscilloscope98]

Hi @johnysh,

Currently, we have not natively supported time consumption msg about first token and rest tokens latency in log for Langchain-Chatchat. However, you could do that with the help of ipex-llm benchmark tool.

To use benchmark tool in Langchain-Chatchat:

1. Put benchmark_util.py in your conda env for langchain-chatchat:

   The path to put the script should be like (taking linux os as an example): /home/<user_name>/<anaconda3 or miniconda3>/envs/<your conda env name>/lib/python3.11/site-packages/ipex_llm/serving/fastchat/benchmark_util.py

2. In /home/<user_name>/<anaconda3 or miniconda3>/envs/<your conda env name>/lib/python3.11/site-packages/ipex_llm/serving/fastchat/ipex_llm_worker.py, add BenchmarkWrapper to model:

   That is, change the code here to:

       self.model, self.tokenizer = load_model(
           model_path, device, self.load_in_low_bit, trust_remote_code
       )
   
       from .benchmark_util import BenchmarkWrapper
       self.model = BenchmarkWrapper(self.model)

3. In /home/<user_name>/<anaconda3 or miniconda3>/envs/<your conda env name>/lib/python3.11/site-packages/ipex_llm/serving/fastchat/ipex_llm_worker.py, add print message for 1st and rest token latency.

   That is, change the code here to:

       print(f"First token latency (s): {self.model.first_cost}", flush=True)
       print(f"Rest token latency (s): {self.model.rest_cost_mean}", flush=True)
   
       yield json.dumps(json_output).encode() + b"\0"

Please let us know for any further problems :)