search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
19 stars 26 forks source link

use global variables instead of strings in openai backend #19

Closed davidschlangen closed 5 months ago

davidschlangen commented 7 months ago

https://github.com/clp-research/clembench/blob/213f184e8dfbcc107a7dc868b6e1c52ef09d4eba/backends/openai_api.py#L28

Should this not use SUPPORTED_MODELS rather than listing the model names again? Adding a new model would mean adding this information at more than one place, I think; would be better if there's only one.

davidschlangen commented 7 months ago

Ah hang on, it's a subset of SUPPORTED_MODELS. But in any case, should then get a global variable as well, so that all this information is in the header.

davidschlangen commented 5 months ago

Whatever was the case here, this is going to be addressed by #26 . Closing for now.

phisad commented 5 months ago

Btw. the "model_type" might be noted in the OpenAI ModelSpecs of the registiry e.g. {"model_type":"chat"} This will be then available via the ModelSpec.opts attribute