Inconsistent MSA representation

facebookresearch / esm

Evolutionary Scale Modeling (esm): Pretrained language models for proteins

MIT License

3.16k stars 627 forks source link

Inconsistent MSA representation #178

Closed valentas1 closed 2 years ago

valentas1 commented 2 years ago

Hi,

I am trying to get a representation of an MSA of my protein with the esm_msa1b_t12_100M_UR50S model. However, the representation I am receiving is inconsistent - calling the same function with the same data a few times I am getting slightly different representations. Do you know where this inconsistency could come from?

tomsercu commented 2 years ago

Most likely culprit is dropout. Dropout is disabled when the model is in eval mode, eg see extract.py#L64 model.eval(). On GPU device there's some non-determinism expected, which can lead to slightly different results. The output should still pass torch.allclose check though.

tomsercu commented 2 years ago

Feel free to reopen if this didn't answer your question. README was also just updated to default to model.eval() to avoid this confusion.

valentas1 commented 2 years ago

Thank you for the very quick answer, setting model.eval() actually worked.