Fail to load random states from checkpoints saved

THUDM / SwissArmyTransformer

SwissArmyTransformer is a flexible and powerful library to develop your own Transformer variants.

https://THUDM.github.io/SwissArmyTransformer

Apache License 2.0

951 stars 90 forks source link

Fail to load random states from checkpoints saved #137

Open minkowski0125 opened 11 months ago

minkowski0125 commented 11 months ago

I loaded checkpoint to continue pretrain and found the following error.

[2023-09-25 15:53:27,994] [INFO] [RANK 0] Unable to load optimizer from checkpoint <ckpt_path>, exiting. Specify --no-load-rng or --finetune to prevent attempting to load the random state.

checked to find there is no 'rng_tracker_states' item saved in function save_checkpoint but loaded in load_checkpoint (sat/training/model_io.py).

DongnuanCai commented 5 months ago

Has this issue been resolved?

elesun2018 commented 4 months ago

请问optimizer是如何保存的，继续训练时，没有找到加载的地方