Closed lin88lin8850 closed 1 year ago
Hi, thanks for using Merak, could you provides more parameter settings? One possible solution might be using a larger number of microbatch (gradient_accumulation_steps in training args).
Hi, thanks for using Merak, could you provides more parameter settings? One possible solution might be using a larger number of microbatch (gradient_accumulation_steps in training args).
MerakArguments(
_n_gpu=8,
activation_checkpoint_ratio=None,
activation_checkpointing=True,
adafactor=False,
bf16=False,
bf16_full_eval=False,
cache_name=None,
cache_sharding=False,
checkpoint_num_layers=1,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=False,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
finetune=False,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_predivide_factor=1.0,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=
@lin88lin8850 The number of microbatch (gradient_accumulation_steps in training args) is 1 in your setting, which make a inefficient pipeline parallelism, please try to set number of microbatch (gradient_accumulation_steps in training args) to at least the number of pipeline stages. We will add related infos in next commit.
@lin88lin8850 The number of microbatch (gradient_accumulation_steps in training args) is 1 in your setting, which make a inefficient pipeline parallelism, please try to set number of microbatch (gradient_accumulation_steps in training args) to at least the number of pipeline stages. We will add related infos in next commit.
hi, lucas, bug still exists! when I set gradient_accumulation_steps = 2, which is same to the pipeline stages
@lucasleesw when I set gradient_accumulation_steps = 4 (dp=tp=pp=2), it's ok please add more information in doc in your convenience, thanks!
some parameters: num_gpu=8 dp=tp=pp=2 train_schedule="shifted_critical_path"