clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
24 stars 34 forks source link

[games] make separation between games and worlds clearer #133

Open davidschlangen opened 2 weeks ago

davidschlangen commented 2 weeks ago

Ideally, we will reach a state where, at least for some games, there is a clearer separation between world (the provider of objects, actions for operating on them, methods for making observations, and tests of (goal) conditions) and games also in the code.

E.g., there could be a structure like this:

worlds/
-- MapWorld/
games/
-- ExploreMapWorld/
-- FindOnMap/

where both games import the same world. (And later, there could also be situations where the same game can optionally be played in different worlds, or worlds with different renderings (like the text-only and the image-only version of MapWorld games.)

Gnurro commented 2 weeks ago

In my current AdventureGame version, that separation is made via instances. Each instance contains a list of definitions it uses, essentially covering all but the last element of world above. Each also contains a prompt for the task (differ between my two variants), goal conditions and initial world state (containing room layout, for example).

What would games uniquely contain that requires them to be as separated as that structure above suggests?

davidschlangen commented 1 week ago

I find it easy to imagine games that are sufficiently different so as to it being inconvenient to express their difference through instances, yet using the same world. E.g., the "I spy" game using AI2THOR, vs. a game of "let's make a sandwich together", also using AI2THOR. Both could use the same abstraction layer around the world (imported from ../worlds/AI2THOR), but can have very different game logics (in ../games/).