search

agemagician / ProtTrans

ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models.
Academic Free License v3.0
1.11k stars 153 forks source link

Language modeling heads #7

Closed jaekor91 closed 4 years ago

jaekor91 commented 4 years ago

Thank you for making available these models. Could you also share language modeling classification task heads, e.g., for BERT models?

agemagician commented 4 years ago

Hi @jaekor91 ,

On the Fine Tuning section, you will find examples for fine-tuning ProtBert model for a different tasks.

You simply need to run it on Google Colab, then you can download the model.

Is this what you are looking for?

agemagician commented 4 years ago

I have also started to add new model heads for classification tasks: https://github.com/agemagician/ProtTrans/blob/master/Prediction/ProtBert_BFD_Predict_MS.ipynb

I will close this issue and feel free to open it if the above comments didn't answer your question.

jaekor91 commented 4 years ago

@agemagician Sorry for the late reply. I was referring to amino acid classification head that is used to turn contextual embeddings into probs over AAs.

agemagician commented 4 years ago

Our models support all huggingface model heads, if what you are looking there, then it is supported, otherwise, not. You can find the list of heads for each model in huggingface website, for example, this is a list of all Bert heads which is supported: https://huggingface.co/transformers/model_doc/bert.html#