Closed ConnorJL closed 1 year ago
I spoke with Connor earlier today, and I think I can give this one a try.
@jmerizia Opened a repo for this project: https://github.com/EleutherAI/attention-ensemble
Jacob did some good work here, but ultimately the FF layers in actual transformers ruin the abstraction as they "mix" all the different paths, and we haven't come up with any good idea of how to fix this problem. If anyone can come up with a new theoretical abstraction to generalize this analysis to full transformers, I'd love to hear it, otherwise this project is probably not possible.
Jacob did some good work here, but ultimately the FF layers in actual transformers ruin the abstraction as they "mix" all the different paths, and we haven't come up with any good idea of how to fix this problem. If anyone can come up with a new theoretical abstraction to generalize this analysis to full transformers, I'd love to hear it, otherwise this project is probably not possible.
This seems like a sufficiently noteworthy negative result to merit a blog post