search

DeepChainBio / bio-transformers

bio-transformers is a wrapper on top of the ESM/Protbert model, trained on millions on proteins and used to predict embeddings.
https://bio-transformers.readthedocs.io/en/latest/getting_started/install.html
Apache License 2.0
143 stars 31 forks source link

embedding in ESM is not extracted from the last layer #12

Closed mathpluscode closed 3 years ago

mathpluscode commented 3 years ago

Our current code is:

https://github.com/DeepChainBio/bio-transformers/blob/main/biotransformers/wrappers/esm_wrappers.py#L143

But if you check the ESM extract code:

https://github.com/facebookresearch/esm/blob/master/extract.py#L80

You can see if i = -1, then (i + model.num_layers + 1) % (model.num_layers + 1) = model.num_layers

delfosseaurelien commented 3 years ago

Thanks for the fix!