search

the-crypt-keeper / can-ai-code

Self-evaluating interview for AI coders
https://huggingface.co/spaces/mike-ravkine/can-ai-code-results
MIT License
524 stars 30 forks source link

Evaluate google/gemma-2-27b-it #209

Closed the-crypt-keeper closed 3 months ago

the-crypt-keeper commented 3 months ago

Appears there's some lingering issues with the 27b, I was unable to get it to produce anything other then nonsense. Revisit later.

the-crypt-keeper commented 3 months ago

See this reddit thread the trick is attn_implementation='eager'

27B FP16 debuts at number 1 on the leaderboard 🏆