search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
5.82k stars 1.55k forks source link

Evaluation for MegatronT5 Model #1907

Closed wangyanbao666 closed 2 days ago

wangyanbao666 commented 1 month ago

Hi I have noticed that there are existing support for some nemo models. But it does not seem that there is a support for MegatronT5 Model. Anyone has ideas how to evaluate this model?

haileyschoelkopf commented 1 month ago

Hi! We don't currently support this, though would welcome contributions adding it.

Perhaps @sergiopperez has input on how simple it would be to extend the current NeMo integration into Megatron-T5?

wangyanbao666 commented 1 month ago

Hi any updates on how to evaluate T5? Could you please give some clue on how to extend to Megatron-T5?

StellaAthena commented 1 month ago

Hi any updates on how to evaluate T5? Could you please give some clue on how to extend to Megatron-T5?

If you are interested in evaluating a pretrained model such as NeMo Megatron T5 that has been released on Hugging Face, the Hugging Face version is compatible with this library. However we still lack support for models native to the Megatron library.

haileyschoelkopf commented 2 days ago

Closing as NeMo Megatron-T5 support won't be prioritized, though we'd welcome a contribution for it.