Closed hetian127 closed 3 months ago
看起来是/share/home/hetian/dataset/open_qa.jsonl文件不对,不是正常的openqa格式,导致解析失败了,https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl line by line: {"question":"比斯巨兽的皮干什么用的?多少钱? ","human_answers":["做史诗装备用的~~把比斯巨兽(黑石塔上层的一个BOSS)的皮拨了(很小几率出比斯巨兽皮~大部分是厚皮)不管是不是出比斯巨兽的皮都会出现一个地精>(从巨兽肚子中出来的)会给你一个任务这个任务要去冬泉交(交给冬泉瑟银会的NPC)~交完之后他身后有本书上面写着3件史诗装备所需要的原料(都需要比斯巨兽皮和其他物品)\r\n忘记说多少钱了~我们服是600G~700G不一样>的服价格也不一样~"],"chatgpt_answers":["比斯巨兽是什么?我不知道这是什么东西。您可以告诉我更多关于比斯巨兽的信息吗?我将尽力帮助您。"]}
看起来是/share/home/hetian/dataset/open_qa.jsonl文件不对,不是正常的openqa格式,导致解析失败了,https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl line by line: {"question":"比斯巨兽的皮干什么用的?多少钱? ","human_answers":["做史诗装备用的~~把比斯巨兽(黑石塔上层的一个BOSS)的皮拨了(很小几率出比斯巨兽皮~大部分是厚皮)不管是不是出比斯巨兽的皮都会出现一个地精>(从巨兽肚子中出来的)会给你一个任务这个任务要去冬泉交(交给冬泉瑟银会的NPC)~交完之后他身后有本书上面写着3件史诗装备所需要的原料(都需要比斯巨兽皮和其他物品)\r\n忘记说多少钱了~我们服是600G~700G不一样>的服价格也不一样~"],"chatgpt_answers":["比斯巨兽是什么?我不知道这是什么东西。您可以告诉我更多关于比斯巨兽的信息吗?我将尽力帮助您。"]}
非常感谢,我之前用wget https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl 得到的文件是不对的。这个问题已经解决
环境: llmuses 0.4.0
我在本地起了一个基于llama3 的vllm api server,准备用llmuses perf命令来做benchmark,命令如下:
(eval-scope) xxx@host:~$ llmuses perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 4 --model 'llama3' --log-every-n-query 10 --read-timeout=120 --dataset-path '/share/home/hetian/dataset/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --n-choices 3 --stop-token-ids 128001 128009 --dataset openqa --debug
报错如下:
Save the result to : ./llama3_benchmark_2024_07_02_15_43_22_647227.db Traceback (most recent call last): File "/share/home/xxx/miniconda3/envs/eval-scope/bin/llmuses", line 8, in
sys.exit(run_cmd())
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/cli.py", line 19, in run_cmd
cmd.execute()
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/start_perf.py", line 33, in execute
run_perf_benchmark(self.args)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 654, in run_perf_benchmark
asyncio.run(benchmark(args))
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 597, in benchmark
expected_number_of_queries = await dispatch_task # wait for dispatch task complete
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 231, in dispatch_requests_worker
for messages in message_generator:
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/datasets/openqa.py", line 19, in build_messages
item = json.loads(item)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
请问如何处理,多谢了