Closed Longyichen closed 9 months ago
Hi! Our initial development was done for Pythia series of models and we pruned from Pythia-410m to a 160M model with the same recipe we did for Llama models, and we created a better Sheared-Pythia-160m.
I didn't have the time and bandwidth to sort out that part of the code for now -- but would appreciate it if you could work through the pipeline! I am considering adding support to Pythia and Mistral for the following weeks.
I've encountered the same issue; I tried to modify the Pythia model to a Composer model by imitating your llama rewrite, and although it runs successfully, the output logits are quite different. Could you please expedite the updates for Pythia or Mistral? @xiamengzhou
Pythia helps study the effects of pruning at different LM scales. Noticed that you provided composer_pythia.py, so I tried running this experiment on 1.4b scale pythia. I'm having some problems.
The first is the writing of the pythia_to_composer conversion function for several settings I made. The keymap here is what I inferred based on your composer_pythia.py
In test_composer_hf_equal.py
final result
The structure of the two models looks very different. I don't know if this is an issue with the structural organization in composer_pythia.py. Finally, I tested the output of the two models, and there was a huge difference.
Will you disclose Python-related experimental code? Do you need me to contribute relevant code?