I loaded checkpoint to continue pretrain and found the following error.
[2023-09-25 15:53:27,994] [INFO] [RANK 0] Unable to load optimizer from checkpoint <ckpt_path>, exiting. Specify --no-load-rng or --finetune to prevent attempting to load the random state.
checked to find there is no 'rng_tracker_states' item saved in function save_checkpoint but loaded in load_checkpoint (sat/training/model_io.py).
I loaded checkpoint to continue pretrain and found the following error.
checked to find there is no 'rng_tracker_states' item saved in function
save_checkpoint
but loaded inload_checkpoint
(sat/training/model_io.py
).