Running inference using evaluation scripts

[srikhetramohanty]

Hi, I want to run inference on the trained models (SFT, distilled) to evaluate them with further metrics. Do you happen to have any script specific to inference only? Since I don't see it explicitly in the repo, I am consuming the evaluation script to do the above but I am running into issues as follows :

1. The scripts/tinyllama/run_eval_lora.sh file mentions a working directory, model path, checkpoint path separately, any example of how this is to be found out from the output folder?
2. Getting an error while putting the following items above :

   awk: cmd. line:1: (FILENAME=- FNR=1) fatal: attempt to access field -2
   bash: /home/ubuntu/miniconda3/lib/libtinfo.so.6: no version information available (required by bash)

   WORK_DIR="/DSKD" MODEL_PATH="outputs/mistral/mistral-7b-v0.1" LORA_ADAPTER_PATH = epoch10_step14300_loss3.2432_rougel34.5630 (as argument in cmd)

Thanks in advance.

[songmzhang]

Hi, after running run_eval.sh or run_eval_lora.sh, you will get answers.jsonl and preds.txt in a RESULT_PATH (e.g., ./results/mistral/mistral-7b-v0.1/sft/criterion=cross_entropy_...), which contain the results generated by the trained models and you can extract them for further evaluation. (P.S., by default, there are five test sets in the run_eval.sh and the above two files will only record the results of the last test set.)

About the error, when you pass LORA_ADAPTER_PATH to the script, do you use the absolute directory? If not, please try it.

It seems that the use of our scripts is still kind of confusing. Thus, we will update README later to add a use case for you to better understand it.

[srikhetramohanty]

Hi, thank you for the clarification. I was passing the relative path. It got solved with absolute directory path.