search

axolotl-ai-cloud / axolotl

Go ahead and axolotl questions
https://axolotl-ai-cloud.github.io/axolotl/
Apache License 2.0
7.48k stars 808 forks source link

[Feature] Allow passing file to inference on #95

Open NanoCode012 opened 1 year ago

NanoCode012 commented 1 year ago

Problem

It may be necessary to repeat the same questions across many experiments. It is time consuming to copy paste line by line.

Feature

Allow passing path to jsonl file or similar that can be read and ran through the model then output to a results file

theobjectivedad commented 1 year ago

+1 to this, it would be so useful to leverage the dataset prompt generation logic to benchmark models on holdout dataset(s).

NanoCode012 commented 1 year ago

Hey @theobjectivedad , I have some unclean code which I just pushed to show you how this can be done.

https://github.com/OpenAccess-AI-Collective/axolotl/compare/main...NanoCode012:axolotl:feat/inference-file?expand=1

It uses --inference_file flag.

I will not merge this to current main as it was just a hack for my personal use.

theobjectivedad commented 1 year ago

Giant thank you @NanoCode012 ... I'll take a look!

theobjectivedad commented 1 year ago

@NanoCode012 - do you have plans to contribute this? No rush but if not, I may take a shot at it.

NanoCode012 commented 1 year ago

Hey @theobjectivedad , go for it! Just make sure that the new feature does not lose the current inference method.