search

facebookresearch / esm

Evolutionary Scale Modeling (esm): Pretrained language models for proteins
MIT License
3.16k stars 627 forks source link

Huggingface Transformers model compatibility? #152

Closed ddofer closed 1 year ago

ddofer commented 2 years ago

Is there any way to directly load the ESM models with the HF Transformers (or similarly compatible) libraries? e.g. with AutoModelForTokenClassification? Thanks!

(I want to do finetuning, and to use additional libraries that support the HF models' format).

tomsercu commented 1 year ago

Very happy to share that - thanks to the amazing work of @Rocketknight1 and huggingface team - this is now possible:

Classification tasks with proteins, just like BERT: https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/protein_language_modeling.ipynb

Fold proteins in Colab or your local GPU and export PDB files: https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/protein_folding.ipynb

Admire7494 commented 9 months ago

Is that work? I found that it's difficult to load ESM1b_650M with half precision in a RTX3090