search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
19 stars 26 forks source link

Double model loading/initialization #85

Closed Gnurro closed 2 months ago

Gnurro commented 2 months ago

With the latest version (as of April 25.), the main log (clembench.log) has double log entries for model loading. This is happening with at least privateshared and wordle. This is likely the cause for memory issues I encountered on a current run of the full benchmark: Inference is very slow (likely due to offloading to RAM/drive), and at least one model crashed with out-of-memory errors. As far as I know the double loading is new, as I did not see the double log entries on earlier runs (which also were as fast as the hardware makes me expect). I don't have the time to properly investigate this for now, but I suspect this is due to some change in Player/game initialization, with two different Player (subclass) instances each loading the same model. Instead all Players with the same model should use one single instance of Model.

Gnurro commented 2 months ago

The performance issues might have been a fluke, and the misleading logging should be fixed with https://github.com/clp-research/clembench/commit/afccf94fece3e57735b21d7b56635cbec2b15b1e . It might be good if someone double-checks this, but I think the issue can be closed.

davidschlangen commented 2 months ago

Have you observed any more performance problems, @Gnurro ? If not, we can close -- @phisad ?

phisad commented 2 months ago

Can be closed as stated above