liuyhwangyh
commented
2 weeks ago 看起来是/share/home/hetian/dataset/open_qa.jsonl文件不对，不是正常的openqa格式，导致解析失败了，https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl line by line: {"question":"比斯巨兽的皮干什么用的？多少钱？ ","human_answers":["做史诗装备用的~~把比斯巨兽（黑石塔上层的一个BOSS）的皮拨了（很小几率出比斯巨兽皮~大部分是厚皮）不管是不是出比斯巨兽的皮都会出现一个地精>（从巨兽肚子中出来的）会给你一个任务这个任务要去冬泉交（交给冬泉瑟银会的NPC）~交完之后他身后有本书上面写着3件史诗装备所需要的原料（都需要比斯巨兽皮和其他物品）\r\n忘记说多少钱了~我们服是600G~700G不一样>的服价格也不一样~"],"chatgpt_answers":["比斯巨兽是什么？我不知道这是什么东西。您可以告诉我更多关于比斯巨兽的信息吗？我将尽力帮助您。"]}
hetian127
环境： llmuses 0.4.0
我在本地起了一个基于llama3 的vllm api server，准备用llmuses perf命令来做benchmark，命令如下：
(eval-scope) xxx@host:~$ llmuses perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 4 --model 'llama3' --log-every-n-query 10 --read-timeout=120 --dataset-path '/share/home/hetian/dataset/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --n-choices 3 --stop-token-ids 128001 128009 --dataset openqa --debug
报错如下：
Save the result to : ./llama3_benchmark_2024_07_02_15_43_22_647227.db Traceback (most recent call last): File "/share/home/xxx/miniconda3/envs/eval-scope/bin/llmuses", line 8, in
sys.exit(run_cmd())
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/cli.py", line 19, in run_cmd
cmd.execute()
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/start_perf.py", line 33, in execute
run_perf_benchmark(self.args)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 654, in run_perf_benchmark
asyncio.run(benchmark(args))
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 597, in benchmark
expected_number_of_queries = await dispatch_task # wait for dispatch task complete
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 231, in dispatch_requests_worker
for messages in message_generator:
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/datasets/openqa.py", line 19, in build_messages
item = json.loads(item)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
请问如何处理，多谢了