Open vlsav opened 3 weeks ago
tastelikefeet
commented
2 weeks ago Which version of code were you using? In a short period, the eval_token argument lost its functionality, but we have fix this bug Can you try the latest main branch code please?
vlsav
commented
2 weeks ago That was with 2.1.1.post2. I tried to install latest 2.2.1, but is has dependency on transformers 4.41.2, which I can't use because vllm 0.5.1 requires at least 4.42 So can't check it now
Describe the feature Typically OpenAI API's are protected with some API_KEY, so to run requests to such API's it is necessary to specify API_KEY in a call. Is it planned to add such functionality and, if yes, when it can be expected?
Paste any useful information There is a key "eval_token" which looks like reserved for such use (API_KEY), but it is not utilized (not used in source code) for this purpose. But for now it is not working, see logs below.
Additional context I defined eval_url and eval_token and launched swift eval, but it is not working, as without API_KEY vllm doesn't allow acess ("Unauthorized") so evaluation cannot proceed further.
Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 72, in from_dict value = get_default_value_for_field(field, field_type) File "/home//.local/lib/python3.10/site-packages/dacite/dataclasses.py", line 21, in get_default_value_for_field raise DefaultValueNotFoundError() dacite.dataclasses.DefaultValueNotFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/swift/cli/eval.py", line 5, in
eval_main()
File "/home/ /.local/lib/python3.10/site-packages/swift/utils/run_utils.py", line 27, in x_main
result = llm_x(args, kwargs)
File "/home/**/.local/lib/python3.10/site-packages/swift/llm/eval.py", line 185, in llm_eval
return run_eval_single_model(args)
File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 161, in run_eval_single_model
run_task(task_cfg=task_configs)
File "/home//.local/lib/python3.10/site-packages/llmuses/run.py", line 156, in run_task
eval_results.append(run_task(one_task_cfg))
File "/home//.local/lib/python3.10/site-packages/llmuses/run.py", line 314, in run_task
res_dict: dict = evaluator.eval(infer_cfg=infer_cfg, debug=debug)
File "/home//.local/lib/python3.10/site-packages/llmuses/evaluator/evaluator.py", line 475, in eval
answers_list: list = self.get_answers(subset_name=subset_name,
File "/home//.local/lib/python3.10/site-packages/llmuses/evaluator/evaluator.py", line 197, in get_answers
resp_answers_list: List[Dict[str, Any]] = self.model_adapter.predict(inputs=prompts_list,
File "/home/*/.local/lib/python3.10/site-packages/llmuses/models/model_adapter.py", line 585, in predict
return self.custom_model.predict(prompts=in_prompts, *kwargs)
File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 85, in predict
response_list = asyncio.run(self.call_openai_batched(prompts, request_config))
File "/opt/llm/miniconda3/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/llm/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 64, in call_openai_batched
response, i = await coro
File "/opt/llm/miniconda3/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 40, in _call_openai
resp = await inference_client_async(
File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 269, in inference_client_async
url, data, is_chat_request = _pre_inference_client(
File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 155, in _pre_inference_client
model_list = get_model_list_client(host, port, kwargs)
File "/home/**/.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 30, in get_model_list_client
return from_dict(ModelList, resp_obj)
File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 76, in from_dict
raise MissingValueError(field.name)
dacite.exceptions.MissingValueError: missing value for field "data"
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<EvalModel._call_openai() done, defined at /home//.local/lib/python3.10/site-packages/swift/llm/eval.py:36> exception=MissingValueError()>
Traceback (most recent call last):
File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 72, in from_dict
value = get_default_value_for_field(field, field_type)
File "/home/*****/.local/lib/python3.10/site-packages/dacite/dataclasses.py", line 21, in get_default_value_for_field
raise DefaultValueNotFoundError()
dacite.dataclasses.DefaultValueNotFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 40, in _call_openai resp = await inference_client_async( File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 269, in inference_client_async url, data, is_chat_request = _pre_inference_client( File "/home/*/.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 155, in _pre_inference_client model_list = get_model_list_client(host, port, *kwargs) File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 30, in get_model_list_client return from_dict(ModelList, resp_obj) File "/home/*****/.local/lib/python3.10/site-packages/dacite/core.py", line 76, in from_dict raise MissingValueError(field.name) dacite.exceptions.MissingValueError: missing value for field "data"
Task exception was never retrieved future: <Task finished name='Task-4' coro=<EvalModel._call_openai() done, defined at /home//.local/lib/python3.10/site-packages/swift/llm/eval.py:36> exception=MissingValueError()> Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 72, in from_dict value = get_default_value_for_field(field, field_type) File "/home/*****/.local/lib/python3.10/site-packages/dacite/dataclasses.py", line 21, in get_default_value_for_field raise DefaultValueNotFoundError() dacite.dataclasses.DefaultValueNotFoundError
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 40, in _call_openai resp = await inference_client_async( File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 269, in inference_client_async url, data, is_chat_request = _pre_inference_client( File "/home/*/.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 155, in _pre_inference_client model_list = get_model_list_client(host, port, *kwargs) File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 30, in get_model_list_client return from_dict(ModelList, resp_obj) File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 76, in from_dict raise MissingValueError(field.name) dacite.exceptions.MissingValueError: missing value for field "data" Task exception was never retrieved future: <Task finished name='Task-5' coro=<EvalModel._call_openai() done, defined at /home//.local/lib/python3.10/site-packages/swift/llm/eval.py:36> exception=MissingValueError()> Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 72, in from_dict value = get_default_value_for_field(field, field_type) File "/home//.local/lib/python3.10/site-packages/dacite/dataclasses.py", line 21, in get_default_value_for_field raise DefaultValueNotFoundError() dacite.dataclasses.DefaultValueNotFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 40, in _call_openai resp = await inference_client_async( File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 269, in inference_client_async url, data, is_chat_request = _pre_inference_client( File "/home/*/.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 155, in _pre_inference_client model_list = get_model_list_client(host, port, *kwargs) File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 30, in get_model_list_client return from_dict(ModelList, resp_obj) File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 76, in from_dict raise MissingValueError(field.name) dacite.exceptions.MissingValueError: missing value for field "data" Task exception was never retrieved future: <Task finished name='Task-6' coro=<EvalModel._call_openai() done, defined at /home//.local/lib/python3.10/site-packages/swift/llm/eval.py:36> exception=MissingValueError()> Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/dacite/core.py", line 72, in from_dict value = get_default_value_for_field(field, field_type) File "/home//.local/lib/python3.10/site-packages/dacite/dataclasses.py", line 21, in get_default_value_for_field raise DefaultValueNotFoundError() dacite.dataclasses.DefaultValueNotFoundError
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home//.local/lib/python3.10/site-packages/swift/llm/eval.py", line 40, in _call_openai resp = await inference_client_async( File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 269, in inference_client_async url, data, is_chat_request = _pre_inference_client( File "/home/*/.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 155, in _pre_inference_client model_list = get_model_list_client(host, port, *kwargs) File "/home//.local/lib/python3.10/site-packages/swift/llm/utils/client_utils.py", line 30, in get_model_list_client return from_dict(ModelList, resp_obj) File "/home/*****/.local/lib/python3.10/site-packages/dacite/core.py", line 76, in from_dict raise MissingValueError(field.name) dacite.exceptions.MissingValueError: missing value for field "data"