vienneraphael / layton-eval

2 stars 2 forks source link

Create LLM answer structuration (!) not OpenAI #2

Closed vienneraphael closed 9 months ago

vienneraphael commented 10 months ago

NOTE: I think it'd be good that we are cautious not to use the OpenAI API in order to avoid biasing our benchmark by feeding GPT3.5 answers, in case OpenAI ever uses our samples to re-train their models.

I suggest we use an open-source model with an inference api platform like Together AI.

=> Write a script to structure answers from riddles with any model available on Together AI (they offer free credits!)

Can be tackled with #1 using LangChain