Possible checkpoint loss in TAPE pretraining scripts

THUDM / ProteinLM

Protein Language Model

Apache License 2.0

112 stars 20 forks source link

Possible checkpoint loss in TAPE pretraining scripts #1

Closed Yijia-Xiao closed 3 years ago

Yijia-Xiao commented 3 years ago

In pretraining scripts for TAPE mode, there are two commands removing all contents in checkpoint folder, this may cause lost of previous checkpoints.

https://github.com/THUDM/ProteinLM/blob/main/pretrain/examples/pretrain_tape_distributed.sh#L17
https://github.com/THUDM/ProteinLM/blob/main/pretrain/examples/pretrain_tape.sh#L8