Open noanabeshima opened 1 month ago
It would be quite nice to add this using the new model adder that @ArthurZucker has contributed; @ArthurZucker, when back from leave (next week), do you mind sharing with @noanabeshima how to get this done the best way?
Hey! sorry for the delay! Yep, my recommendation is to use the #30868 tool to isolate the changes as much as possible 🤗
hi @ArthurZucker I am new to open-source contribution and I would like to contribute to add this new model to transformer library can you please provide to any reference or previous PRs that were similar to this
Model description
https://github.com/noanabeshima/tiny_model
It's a small language model trained on TinyStories for interpretability with sparse autoencoders and transcoders added. It has no layernorms (this helps with interpretability) which makes it not fit with any existing model architecture in the transformers library. Its architecture is essentially GPT-2's except that it doesn't have layernorms and it has untied embed/deembed.
Open source status
Provide useful links for the implementation
The implementation is here: https://github.com/noanabeshima/tiny_model/blob/main/tiny_model/lm.py
The weights are here: https://huggingface.co/noanabeshima/tiny_model/blob/main/tiny_model.pt
The default config corresponding to the weights is:
I am the author.