mheinzinger
commented
1 year ago Hi; hm, I see multiple options:
- BERT-like-approach: you train a simple NN on top of encoder-outputs and predict amino acid identities. For this, you might want to mask some tokens using the special tokens used by ProtT5 in order to make the task not too simple. Probably you can get away without fine-tuning the whole encoder. Will be computationally light-weight. But after all, this will still not be a real generation approach as it will still require some non-corrupted context from which it will only recover a certain fraction of (of course, if you choose your masks in the right way you can sequentially generate tokens but this might give worse performance than a decoder-approach).
- Fine-tune approach: we trained only on span-generation (after all, we used span-length=1 which is similar to denoising/masked-language-modeling). So ProtT5 can give you logits for masked positions but it might not give very good results for the "traditional"/GPT-like generation tasks. For this to work well, you probably want to fine-tune whole ProtT5. This will require some compute.
- Only train a decoder model: you could try to only train a decoder given the ProtT5 encoder-outputs. But for this, I would probably rather go for the option above and fine-tune ProtT5 right away.
gitwangxf
Hi! I’ m trying to use the encoder and decoder of ProtT5 seperately, but failed to get correct protein sequences passing only the encoder_outputs to T5ForConditionalGeneration model as well as model.generate(). I was wondering if there is a way to generate protein sequences from the encoder outputs only? Thanks for your help!