Open Richar-Du opened 1 month ago
When I try to resume a checkpoint from last training, it raises an error:
[rank12]: Traceback (most recent call last): [rank12]: File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/ST-LLM-temp/stllm/train/train_hf.py", line 278, in <module> [rank12]: train() [rank12]: File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/ST-LLM-temp/stllm/train/train_hf.py", line 267, in train [rank12]: trainer.train(resume_from_checkpoint=True) [rank12]: File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train [rank12]: return inner_training_loop( [rank12]: File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/transformers/trainer.py", line 1893, in _inner_training_loop [rank12]: epoch_iterator = skip_first_batches(epoch_iterator, steps_trained_in_current_epoch) [rank12]: File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/accelerate/data_loader.py", line 1086, in skip_first_batches [rank12]: dataset = dataloader.dataset [rank12]: AttributeError: 'MetaLoader' object has no attribute 'dataset'
Sorry, our code currently only supports resume from epoch, not resume from step. Resuming from step cannot retrieve the historical order of the dataset.
When I try to resume a checkpoint from last training, it raises an error: