search

clp-research / clembench

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
MIT License
19 stars 26 forks source link

double check huggingface backend, apparently does not get passed temperature parameter correctly #15

Closed davidschlangen closed 3 months ago

davidschlangen commented 8 months ago

(As reported by Jonathan) - backend complains about temperature parameter. The other backends do get (or at least are supposed) to get this parameter when called, so need to go back and check this.

Gnurro commented 8 months ago

In theory the current code should work as intended - I will test this along with testing the local models and report back if there's an actual issue. This may just be due to the transformers generate method being rather bloated and allowing too many potentially conflicting arguments to be passed. In that case, simply turning off the transformers warnings would 'solve' the issue. See https://github.com/huggingface/transformers/issues/22405#issuecomment-1485527953 :

We are aware that our .generate() has too many options and too little checks/examples, we are working on it 🤗

Gnurro commented 7 months ago

Temperature is set properly - it's logged now, showing that it's set to 0.0 as intended.