clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
22 stars 31 forks source link

make it possible to determine programmatically which `instances.json`is being used for a run #12

Closed davidschlangen closed 7 months ago

davidschlangen commented 11 months ago

A version of the clembench benchmark is determined by a) the set of games that is run and b) the instances for each game. At the moment, this information is distributed over various places:

Assuming that the set of games constitutes the major version number, and the set of instances the minor version number, it would be good if this information could be consolidated.

One way of doing this would be to make it possible for different instances.json to live in in/, and make them addressable via the version number. At the moment, AFAIK this is not exposed via the code.

phisad commented 7 months ago

Duplicated of https://github.com/clp-research/clembench/issues/12

davidschlangen commented 7 months ago

Duplicate of #36 , in a way...