search

BoundaryML / baml

BAML is a templating language to write typed LLM functions. Check out the promptfiddle.com playground
https://docs.boundaryml.com
Apache License 2.0
428 stars 13 forks source link

Few-shot examples support? #756

Open mbbyn opened 3 days ago

mbbyn commented 3 days ago

Thanks for this refreshing take on LLM prompt generation and evaluation, it's very promising.

I was wondering if few-shot examples should have their own first-class support in BAML due to their powerful influence on teaching the model both syntax and semantics simultaneously?

I acknowledge that it's currently possible to use Jijna + messages to inject few-shot examples, but seeing first-class support for test cases made me think, why not few-shot? The few-shot examples provided are indeed usable both as test cases and few shot examples in the prompt. In fact, this is what we do for validating our few-shot dataset, is by doing a k-fold split where we pass the train split as few-shot examples, and the test split as test cases.

mbbyn commented 3 days ago

(I combed through the docs to see if there is a way to "instantiate" BAML classes into an array and return them from "FewShotExamples()" function, but not sure yet whether it's possible or not. Sorry if it is and I missed it.)

aaronvg commented 3 days ago

Ignore the bot, it s gone crazy. We dont have a specific syntax yet for it, but it is on our roadmap. We currently dont have a timeline yet unfortunately!

Thanks for the suggestion!

aaronvg commented 3 days ago

The way it can be done at the moment is to pass the examples as an argument to the function and render the examples in the prompt.

So they are basically constructed on the python or Ts side