tomsercu
commented
2 years ago Thanks for getting in touch!
whether MSA transformer model is finetuned while training the downstream deep residual network
No, we don't finetune MSA Transformer here, the model is kept frozen.
how MSA transformer incorporate its representation into the downstream head based on the Netsurf architecture
we extract sequence representations corresponding to the main sequence (query sequence of the MSA) only. so if emb is [M x L x d] we take emb[0]. Another logical alternative, which performed similar but a little worse, is to average over the msa so use emb.mean(0).
No feature combination done here with the HMM features.
Yijia-Xiao
Hi, I have two questions regarding the details of downstream tasks (section 4.2. Supervised Contact Prediction & section 4.3. Secondary Structure Prediction) in MSA Transformer For
supervised contact prediction, the paper mentionedwe train a deep residual network. I'm wondering whether the MSA transformer model is finetuned while training the downstreamdeep residual network. Forsecondary structure prediction, the MSA paper mentionedwe train a state-of-the-art downstream head based on the Netsurf architecture (Klausenet al., 2019). The downstream model is trained to predict 8-class secondary structure from the pretrained representations; then I referred to the paper ofNetsurf, which has the architecture description as given below split line.I am wondering how MSA transformer incorporate its representation into the
downstream head based on the Netsurf architecture, and whetherHMM profilesis used in SS8 prediction? I would appreciate it if you could provide more details about the structure used in SS8.Thank you!
-- split line