search

0-5788719150923125 / vtx

an experiment
Other
7 stars 1 forks source link

Implement a test step #7

Open Vectorrent opened 10 months ago

Vectorrent commented 10 months ago

Probably via the Pythia test harness: https://github.com/EleutherAI/lm-evaluation-harness

Vectorrent commented 6 months ago

There's a basic shell version of the Pythia harness currently working, but we really need to convert to a Python-native version of it. The reason is for models like ModuleFormer; we need to load a custom model architecture, which hooks into the Huggingface Transformers API, before ever calling the harness. I don't think we can do that from the shell (nor are we able to re-use any of our other code).