search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
19 stars 26 forks source link

max tokens in backends should not be hardcoded #20

Closed davidschlangen closed 4 months ago

davidschlangen commented 7 months ago

It looks like all backends (or at least the ones I've looked at, openai and huggingface_local), hardcode the maximal number of tokens to be returned in the call to generate(), as a number. At the very least, this should be a global variable at the beginning of the file, but ideally, it should be a parameter that can be set from the outside.

Gnurro commented 7 months ago

The huggingface_local backend does not hardcode max tokens - the generation method can be passed any integer, it just defaults to 100: https://github.com/clp-research/clembench/blob/b978765263064052a4e911a57b8737555ebbd77a/backends/huggingface_local_api.py#L134 The standard value could be set as an attribute like temperature is, if that's the issue. EDIT: This is the same for the llama2_hf_local backend.

davidschlangen commented 7 months ago

True. But the openai one does.

phisad commented 4 months ago

Fixed with #47