Open jianguoz opened 9 months ago
Sorry for the inconvenience, I suspect this might stem from a minor error in my README. Could you please try changing --agent_prompt test_v2
to --agent_prompt test_v1
? This is because test_v1
was the prompt used during the training of this model.
Hi @tangqiaoyu , thanks for your suggestions! I run the code following test_v1. However, I still have below errors:
ValueError: Could not parse LLM output:
errors. See details in
error_out.txt and error_out_2.txt.Instances
and raise errors when we run evaluation.py to eval the generations.
Could you check it? Appreciate your help!Hi @tangqiaoyu, just a kind follow up in case you missed the new comment:). Thanks for your time
I apologize for the delayed response. ValueError: Could not parse LLM output:
is a normal phenomenon, especially with models that haven't been trained on the dataset. As seen in your error_out_2.txt
file, the sequence length is logger than the max length of the model and the model generate a ASSISTANT thought
without a ASSISTANT action
as the end. So the code cannot parse the output.
To address this, I suggest uncommenting the code in generation.py at lines 90-92.
Thank you for bringing this to our attention, and please let me know if there are any further issues or questions.
Hi @tangqiaoyu, thanks very much for your detailed reply. I can now run the evaluation successfully after uncommenting lines 90-92. When I evaluate TangQiaoYu/ToolAlpaca-13B
, the most common issue is json formats issue like
log_simulation.txt, i.e., I guess a fine-tuned model or even gpt-3.5-turbo may have difficulties to generate json format. The final output statistics is below
"statistics": {
"num": 100,
"error_num": 10,
"process": {
"Yes": 55,
"No": 35,
"Uncertain": 0
},
"response": {
"Yes": 60,
"No": 30,
"Uncertain": 0
},
"both": 53
},
See the full model generation in eval_ToolAlpaca-13B.json.
I am little bit confused about the definitions of each key. It seems that the scores of "process", "response" and "both" are much lower than the numbers reported in the paper. Could you provide some insights here?
Another question is that I use the suggested test_v1
to evaluate your model. To make the model more robust or improve performance. Should I use both test_v1
and test_v2
if i want to fine-tune the llama2 model in the training data?
Appreciate your time and help!
Hi, @jianguoz, thanks for the further details. Here's my response to each of your questions:
test_v1
because our model was trained with those prompts. It's mainly a format issue, but training with both test_v1
and test_v2
should be fine if you're looking to fine-tune the llama2 model in your training data.@tangqiaoyu Thanks for the insightful comments! It seems that the reproduction produces low results and not sure the potential issues come from the simulator. Do you have updates regarding to the model output details? Thanks:)
@jianguoz Apologies for the delayed response. I have uploaded the GPT-4 evaluation results. Could you please check the model outputs and evaluation results to find the potential issues?
Hi @tangqiaoyu, thanks for your great released work! I am trying to evaluate your checkpoint through:
However, it raises below errors in output (see full error in the attached file):
error.txt
I am following your langchain/transformer version, and sentencepiece==0.1.99, and torch==2.0.0+cu118
Could you please provide some insights here? Appreciate your help