Initialization of paired representation inputs to the ESMFold folding block

[amyxlu]

I'm wondering if you can clarify how the paired representations_z_0 input to the ESMFold folding module is derived. In Lin et al. (2022), Section 1.3 reads:

The second change involves the removal of templates....We simply omit this information, passing instead the attention maps from the language model, as these have been shown to capture structural information well.

In esm/esmfold/v1/esmfold.py#L169, however, it seems that s_z_0 is initialized as zeros, and only res["representations"] is returned from the LM forward pass (line 110). Would you be able to clarify where ESM2 attentions are coerced into paired representation inputs to ESMFold? It'd be especially helpful if you can point to which key in the ESM2 output results dictionary is used.

Thanks!

[nikitos9000]

Hi @amyxlu thanks for noticing that. Indeed, we stopped passing the attention maps from LM to Folding blocks as we were able to maintain the same prediction quality based solely on LM (output) representations, but haven't updated this line in the paper.

[amyxlu]

Hi @nikitos9000, thank you for the response. To clarify, then, for the results that were compared and presented in the paper, were the attention maps / inferred contacts from the LM entirely omitted? I.e. the entire template trunk input into the folding block can be initialized from zero for all sequences, and we rely purely on the recycled paired representation outputs (derived entirely from the LM sequence representations)?

For further development and usage with ESMFold, if you can provide some observations with respect to performance changes observed when actually using the LM attentions / inferred contact maps, that would be really helpful.

Thanks!

[amyxlu]

Addressed in commit ce4c326; closing the issue now. Thanks for the help!