search

facebookresearch / esm

Evolutionary Scale Modeling (esm): Pretrained language models for proteins
MIT License
3.16k stars 627 forks source link

Layer weights for ESM-Fold #262

Closed zhenyuhe00 closed 1 year ago

zhenyuhe00 commented 2 years ago

Hi, Congrats on this series of great work! I'm using your pre-trained model for downstream applications. In the ESM-2 paper, the weighted sum of embeddings from all layers of ESM-2 is fed to the ESM-fold. I wonder if it's enough to only use the embeddings from the last layer of ESM-2, since storing all layers' embedding budget is high. Will the performance drop?

Thanks in advance!

tomsercu commented 2 years ago

I'd expect only a very minor performance drop. If you use fixed weights to make the weighted sum you can avoid additional memory consumption by just summing into a result tensor throughout the network.

zhenyuhe00 commented 2 years ago

Thanks!

zhenyuhe00 commented 2 years ago

I'd expect only a very minor performance drop. If you use fixed weights to make the weighted sum you can avoid additional memory consumption by just summing into a result tensor throughout the network.

I wonder How to get the weights, are they trained when training ESM-Fold, or just manually set to a number? Thanks in advance!

tomsercu commented 2 years ago

will be released with ESMFold

tomsercu commented 1 year ago

Excited to share that ESMFold was released on November 1st!

y-hwang commented 1 year ago

Could you tell us the key under which the layer weights are stored in esmfold_3B_v1.pt ? thank you!