[Project] Are large LMs ensembles of shallow paths?

ConnorJL commented 3 years ago

Project: Are large LMs ensembles of shallow paths?
Project Lead(s): Connor
Project Members: Jacob Merizian
Elevator Pitch: Attention Is Not All You Need shows on page 11 that their (very small) transformer's performance depends mostly on very short paths through the model, making it a kind of "ensemble of shallow attention networks". Can we see this same behavior in large LMs?
Goal Outputs: At least a graph showing how removing paths of various lengths affects performance, and the code to reproduce it. Ideally, we might get some insight into how one could "pull apart" larger networks into smaller networks, which may be a boon to interpretability.
How to Help: DM Connor on Discord.
Desired Support: This experiment can probably be done on colab. If we want to scale up to really large models maybe we need to use some Eleuther GPUs.

jmerizia commented 3 years ago

I spoke with Connor earlier today, and I think I can give this one a try.

StellaAthena commented 3 years ago

@jmerizia Opened a repo for this project: https://github.com/EleutherAI/attention-ensemble

ConnorJL commented 2 years ago

Jacob did some good work here, but ultimately the FF layers in actual transformers ruin the abstraction as they "mix" all the different paths, and we haven't come up with any good idea of how to fix this problem. If anyone can come up with a new theoretical abstraction to generalize this analysis to full transformers, I'd love to hear it, otherwise this project is probably not possible.

StellaAthena commented 2 years ago

Jacob did some good work here, but ultimately the FF layers in actual transformers ruin the abstraction as they "mix" all the different paths, and we haven't come up with any good idea of how to fix this problem. If anyone can come up with a new theoretical abstraction to generalize this analysis to full transformers, I'd love to hear it, otherwise this project is probably not possible.

This seems like a sufficiently noteworthy negative result to merit a blog post

manuelsh commented 2 years ago

Why don't you move the project to "abandoned/dormant" in the board? Is the board active?

EleutherAI / project-menu

[Project] Are large LMs ensembles of shallow paths? #13