search

MadeAgents / Hammer

Hammer: Robust Function-Calling for On-Device Language Models via Function Masking
https://arxiv.org/abs/2410.04587
33 stars 3 forks source link

evaluation issue #6

Open emrecanacikgoz opened 1 week ago

emrecanacikgoz commented 1 week ago

Seems like your evaluation setup is broken, there is no inference since it gives no file error for generated_predictions.jsonl.

I am only evaluating the model, so I don't do trainings. Could you help me to solve the issue:

bash eval.sh MadeAgents/Hammer2.0-7b apibank_l1

Evaluating model: Hammer2.0-7b on dataset: apibank_l1
Traceback (most recent call last):
  File "/home/Hammer/evaluate/evaluate.py", line 38, in <module>
    with open(output_file, 'r') as infile:
         ^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/Hammer/predict/Hammer2.0-7b/apibank_l1/generated_predictions.jsonl'
linqq9 commented 3 days ago

Sorry for the inconvenience. There was an issue with the previous evaluation script (eval.sh) as the inference code was commented out. We have fixed this now. Please check if it can run properly. Moreover, we are currently refactoring our release code, so you can look forward to an improved version. Thank you for your feedback and patience.

emrecanacikgoz commented 3 days ago

@linqq9 thank you for your response. I previously resolved that issue; the current error is unrelated.

It seems there is no generation code to produce the generated_predictions.jsonl file, which is required as an input for evaluation. Could you provide this file and confirm if the entire evaluation pipeline runs successfully on your end?