search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.34k stars 1.68k forks source link

Genericize Arguments #1084

Closed StellaAthena closed 8 months ago

StellaAthena commented 9 months ago

Python allows us to pass arguments to [sub/super]classes elegantly by using the *args and **kwargs arguments. Right now we need to manually add support for every argument not only of our library but also every library that we call to build a LM. For example, we currently don't support inject_fused_attention=False despite the fact that transformers and GPTQ do.

This is best practices in Python and will save us substantial amounts of time and code by offloading work that other libraries are already doing to those libraries.

StellaAthena commented 9 months ago

I would abstractly like to take this on but probably won't have time to do so this month due to NeurIPS + taking time off.

haileyschoelkopf commented 8 months ago

This is addressed in #1096