bytedance / dplm

Official Implemetation of DPLM (ICML'24) - Diffusion Language Models Are Versatile Protein Learners
https://bytedance.github.io/dplm/
Apache License 2.0
76 stars 8 forks source link

mlm training & stage2 training #4

Closed wxy-nlp closed 3 months ago

wxy-nlp commented 3 months ago
  1. add mlm training task and config
  2. add stage2 training code to the models/lm/model_utils.py
  3. add stage2 training config as "dplm_{model_size}_stage2.yaml"
  4. modify the DiffusionProteinLanguageModel.from_pretrained() method, where users can specify the net override config. For example, you load a DPLM model and want to use it for the representation learning, and you want to specify the dropout rate to 0.2, then you can use the following code to load: DiffusionProteinLanguageModel.from_pretrained('airkingbd/dplm_650m', net_override={'hidden_dropout_prob': 0.2})