errors when ran THUDM/chatglm3-6b on this project

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

Apache License 2.0

781 stars 208 forks source link

errors when ran THUDM/chatglm3-6b on this project #182

Closed kimheeyori closed 8 months ago

kimheeyori commented 8 months ago

I ran the THUDM/chatglm3-6b model on this project and found that it reported an error: Error reported when running code: assert self. adding_ Side=="left" AssertionErro, if you have time, take a look. Thank you again.

loubnabnl commented 8 months ago

It seems ChatGLM3-6B requires left padding which wasn't supported in the harness. I implemented a fix in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/184

kimheeyori commented 8 months ago

It seems ChatGLM3-6B requires left padding which wasn't supported in the harness. I implemented a fix in #184

thanks a lot