Fine-tuning protein language model？

[wangleiofficial]

Congratulations on your great work. In my actual experience，Fine-tuning protein language models should lead to better results. Using technologies such as LoRA should not cause a lot of memory consumption. Also, have you tested other protein language models?

[samsledje]

Hi, thanks for your comment! We experimented with directly fine-tuning the full language model, but in our past experience this actually degrades model performance, and as you point out is more memory and compute intensive. Treating the PLM embedding as a fixed representation allows us to pre-compute them to disk and iterate much more quickly on model training.

We haven't yet experimented with more recent methods for lightweight delta tuning such as LoRA, but this is a great idea!

In the supplementary material, we show evaluations comparing ProtBert (which we ended up using in the paper) with ESM and with Prose. We find that each language model is most competitive on different benchmarks, but that ProtBert was most consistently performant. We definitely recommend trying different PLMs, and the ConPLex framework allows for this expansion as PLMa develop in parallel to our work.

[wangleiofficial]

Thank you for bringing me new insights. It is possible that there are significant differences in the tasks related to proteins, and fine-tuning may not always bring benefits. Different PLMs indeed have differences in many tasks, so a large-scale benchmark to comprehensively evaluate different PLMs may be a good idea. I tested our proposed protein language model (https://github.com/ISYSLAB-HUST/ProtFlash, 174M) and found that it performs well in some tasks (TAPE), even better than ESM-1b (650M). LoRA feels like a good method for efficiently fine-tuning PLMs, and in the future, we may build some demos using ProtFlash. If there is an opportunity, I would like to explore some collaboration on this field.

[samsledje]

I'd be interested to read your manuscript on the proposed PLM! Unfortunately I don't think I'll have the bandwidth to take on additional projects this summer, but do feel free to try ConPLex replacing ProtBert with your proposed method, and I'm happy to provide feedback and technical support with the method. If you're still interested in further collaboration beyond, please send an email including my co-authors on this work to see if there is mutual interest? Thanks!