search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
26 stars 34 forks source link

[framework] future-proof: allow any kind of data in conversation #56

Open davidschlangen opened 9 months ago

davidschlangen commented 9 months ago

LMMs integrate images into the conversation. We should generalise this, and let future models / agents pass other objects (like text files, sound, video, camera feeds) into the conversation.

phisad commented 9 months ago

I guess this is related to #39

Gnurro commented 9 months ago

I guess this entails expanding the Player ABC to handle any kind of data in addition to the messages list, and - depending on what the data is used for - the GameMaster ABCs as well. I think a clear delineation between game-side (non-clem Player/GameMaster use) and model-side data would be good as well, as the latter needs backend support for handling the extra data. If there's no clear guidance/examples from the framework for these, I suspect that game creation using arbitrary data could get more messy than it already is...