search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
22 stars 31 forks source link

Make max_new_tokens available in cli script #47

Closed sherzod-hakimov closed 7 months ago

sherzod-hakimov commented 7 months ago

allow the user to set the max_new_token parameter. By default it should be set to 100.

phisad commented 7 months ago

@davidschlangen re issue #26 should this stay a backend property or become a model spec item?

davidschlangen commented 7 months ago

That's related to our discussion about temperature (I believe it was) from a while ago. Under the conception that I mostly have been thinking about, the model spec is more describing the model as a general (the collection of weights and how to run inference on them / access them), and so things like temperature and this do not belong there. But I can see that one can also think about the model spec giving the full specification of a model instance.

Whatever we do, it should be consistent for temperature and max_token.

And my intuition would be that a way to distinguish is "everything that can meaningfully be set differently for each separate call to generate() does not belong to the model spec". But I'm open for discussions here.