Open NanoCode012 opened 1 year ago
+1 to this, it would be so useful to leverage the dataset prompt generation logic to benchmark models on holdout dataset(s).
Hey @theobjectivedad , I have some unclean code which I just pushed to show you how this can be done.
It uses --inference_file
flag.
I will not merge this to current main as it was just a hack for my personal use.
Giant thank you @NanoCode012 ... I'll take a look!
@NanoCode012 - do you have plans to contribute this? No rush but if not, I may take a shot at it.
Hey @theobjectivedad , go for it! Just make sure that the new feature does not lose the current inference method.
Problem
It may be necessary to repeat the same questions across many experiments. It is time consuming to copy paste line by line.
Feature
Allow passing path to jsonl file or similar that can be read and ran through the model then output to a results file