x54-729
commented
1 year ago 非常抱歉这个initization_method_params这个参数已经过时了，您可以再更新一下dev的代码运行试试看
Open skepsun opened 1 year ago
x54-729
commented
1 year ago 非常抱歉这个initization_method_params这个参数已经过时了，您可以再更新一下dev的代码运行试试看
skepsun
commented
1 year ago 还是会报错，传入optimizer的可训练参数是空的，但是前面也对input embedding和lm head设置了requires_grd=True。很奇怪。
[2023-07-20 13:02:58,344] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-20 13:02:58,345] [INFO] [comm.py:594:init_distributed] cdb=None
[2023-07-20 13:02:58,702] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-20 13:02:58,702] [INFO] [comm.py:594:init_distributed] cdb=None
SEED_LAYERS=False BASE_SEED=42 SEED_FN=None
[2023-07-20 13:02:59,072] [INFO] [module.py:358:_partition_layers] Partitioning pipeline stages with method parameters
stage=0 layers=17
0: _inner
1: LlamaLayer
2: LlamaLayer
3: LlamaLayer
4: LlamaLayer
5: LlamaLayer
6: LlamaLayer
7: LlamaLayer
8: LlamaLayer
9: LlamaLayer
10: LlamaLayer
11: LlamaLayer
12: LlamaLayer
13: LlamaLayer
14: LlamaLayer
15: LlamaLayer
16: LlamaLayer
stage=1 layers=18
17: LlamaLayer
18: LlamaLayer
19: LlamaLayer
20: LlamaLayer
21: LlamaLayer
22: LlamaLayer
23: LlamaLayer
24: LlamaLayer
25: LlamaLayer
26: LlamaLayer
27: LlamaLayer
28: LlamaLayer
29: LlamaLayer
30: LlamaLayer
31: LlamaLayer
32: LlamaLayer
33: _inner
34: _inner
loss: GPTLMLoss
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:94 in <module> │
│ │
│ 91 # lr = 0.001, │
│ 92 # clip_grad_norm = 5.0 │
│ 93 # ) │
│ ❱ 94 optimizer = torch.optim.AdamW( │
│ 95 │ filter(lambda p: p.requires_grad, model.parameters()), lr=2e-4) │
│ 96 lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( │
│ 97 │ optimizer, T_max=config.train_epochs * len(train_dataset), eta_min=0) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/adamw.py:50 in __init__ │
│ │
│ 47 │ │ │ differentiable=differentiable, │
│ 48 │ │ │ fused=fused, │
│ 49 │ │ ) │
│ ❱ 50 │ │ super().__init__(params, defaults) │
│ 51 │ │ │
│ 52 │ │ if fused: │
│ 53 │ │ │ if differentiable: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/optim/optimizer.py:187 in __init__ │
│ │
│ 184 │ │ │
│ 185 │ │ param_groups = list(params) │
│ 186 │ │ if len(param_groups) == 0: │
│ ❱ 187 │ │ │ raise ValueError("optimizer got an empty parameter list") │
│ 188 │ │ if not isinstance(param_groups[0], dict): │
│ 189 │ │ │ param_groups = [{'params': param_groups}] │
│ 190 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: optimizer got an empty parameter list
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2165927) of binary: /d1/conda3/envs/scx_llm/bin/python
Traceback (most recent call last):
File "/d1/conda3/envs/scx_llm/bin/torchrun", line 33, in <module>
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
File "/d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 我们发现是这里关于embedding层的替换出了问题，您在pull一下dev分支试一下呢？
skepsun
commented
1 year ago 使用最新的dev分支，报错如下：
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d2/data/chuxiong/collie/examples/further_pretrain_llama/expand_vocab.py:115 in <module> │
│ │
│ 112 │ evaluators=[evaluator] │
│ 113 ) │
│ 114 │
│ ❱ 115 trainer.train() │
│ 116 │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/controller/trainer.py:324 in train │
│ │
│ 321 │ │ │ │ │ self.engine.module.forward_type = "train" │
│ 322 │ │ │ │ if isinstance(self.engine.module, PeftModel) and isinstance(self.engine. │
│ 323 │ │ │ │ │ self.engine.module.get_base_model().forward_type = "train" │
│ ❱ 324 │ │ │ │ with self.monitor as item: │
│ 325 │ │ │ │ │ loss = self.train_fn(self, batch, self.global_batch_idx) │
│ 326 │ │ │ │ │ item.update({"loss": round(loss, 4), │
│ 327 │ │ │ │ │ │ │ │ │ "lr": self.lr, │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/utils/monitor.py:232 in __exit__ │
│ │
│ 229 │ def __exit__(self, exc_type, exc_val, exc_tb): │
│ 230 │ │ for monitor in self.monitors: │
│ 231 │ │ │ monitor.item = self.item │
│ ❱ 232 │ │ │ monitor.__exit__(exc_type, exc_val, exc_tb) │
│ 233 │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/utils/monitor.py:180 in __exit__ │
│ │
│ 177 │ │ return super().__enter__() │
│ 178 │ │
│ 179 │ def __exit__(self, exc_type, exc_val, exc_tb): │
│ ❱ 180 │ │ if self.item["mode"] == "train" and "batch" in self.item.keys(): │
│ 181 │ │ │ self.monitor.write_events([(f"TGS", reduce(lambda x, y: x * y, self.item["ba │
│ 182 │
│ 183 class CPUMemoryMonitor(BaseMonitor): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'mode'
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/controller/trainer.py:325 in train │
│ │
│ 322 │ │ │ │ if isinstance(self.engine.module, PeftModel) and isinstance(self.engine. │
│ 323 │ │ │ │ │ self.engine.module.get_base_model().forward_type = "train" │
│ 324 │ │ │ │ with self.monitor as item: │
│ ❱ 325 │ │ │ │ │ loss = self.train_fn(self, batch, self.global_batch_idx) │
│ 326 │ │ │ │ │ item.update({"loss": round(loss, 4), │
│ 327 │ │ │ │ │ │ │ │ │ "lr": self.lr, │
│ 328 │ │ │ │ │ │ │ │ │ "batch": batch, │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/controller/trainer.py:382 in │
│ train_fn │
│ │
│ 379 │ │ │ │ trainer.engine.module.forward_type = "train" │
│ 380 │ │ │ if isinstance(trainer.engine.module, PeftModel) and isinstance(trainer.engin │
│ 381 │ │ │ │ trainer.engine.module.get_base_model().forward_type = "train" │
│ ❱ 382 │ │ │ loss = trainer.engine.module(**batch)["loss"] │
│ 383 │ │ else: │
│ 384 │ │ │ # concat prompt labels for p-tuning │
│ 385 │ │ │ if trainer.config.peft_config and trainer.config.peft_config.peft_type in [" │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/module.py:549 in forward │
│ │
│ 546 │ │ │ if self.forward_type == "generate": │
│ 547 │ │ │ │ return self.generate_forward(*args, **kwargs) │
│ 548 │ │ │ elif self.forward_type == "train": │
│ ❱ 549 │ │ │ │ return self.train_forward(*args, **kwargs) │
│ 550 │ │ │ elif self.forward_type == "eval": │
│ 551 │ │ │ │ return self.eval_forward(*args, **kwargs) │
│ 552 │ │ │ else: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/module.py:271 in train_forward │
│ │
│ 268 │ │ else: │
│ 269 │ │ │ self._set_past_key_values(past_key_values) │
│ 270 │ │ inputs["labels"] = labels │
│ ❱ 271 │ │ loss = self.engine_container[-1].train_batch(inputs) │
│ 272 │ │ return CausalLMOutputWithPast( │
│ 273 │ │ │ loss=loss, │
│ 274 │ │ │ logits=None, │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/utils/pipeline_engine.py:212 in │
│ train_batch │
│ │
│ 209 │ │ batch = _split_batch(batch, self.train_micro_batch_size_per_gpu(), │
│ 210 │ │ │ │ │ │ │ self.gradient_accumulation_steps()) │
│ 211 │ │ data_iter = iter(batch) │
│ ❱ 212 │ │ result = super().train_batch(data_iter) │
│ 213 │ │ if isinstance(self.module, PipelineModel): │
│ 214 │ │ │ self.module.inner_forward = False │
│ 215 │ │ if isinstance(self.module, PeftModel) and isinstance(self.module.get_base_model( │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/runtime/pipe/engine.py:336 in │
│ train_batch │
│ │
│ 333 │ │ sched = schedule.TrainSchedule(micro_batches=self.micro_batches, │
│ 334 │ │ │ │ │ │ │ │ │ stages=self.num_stages, │
│ 335 │ │ │ │ │ │ │ │ │ stage_id=self.stage_id) │
│ ❱ 336 │ │ self._exec_schedule(sched) │
│ 337 │ │ self.agg_train_loss = self._aggregate_total_loss() │
│ 338 │ │ │
│ 339 │ │ self.timers('train_batch').stop() │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/runtime/pipe/engine.py:1307 in │
│ _exec_schedule │
│ │
│ 1304 │ │ │ │ │
│ 1305 │ │ │ │ # Equivalent to: self._exec_forward_pass(buffer_id=0) │
│ 1306 │ │ │ │ self._exec_instr = MethodType(self._INSTRUCTION_MAP[type(cmd)], self) │
│ ❱ 1307 │ │ │ │ self._exec_instr(**cmd.kwargs) │
│ 1308 │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/utils/pipeline_engine.py:402 in │
│ _exec_forward_pass │
│ │
│ 399 │ │ self._zero_grads(inputs) │
│ 400 │ │ # if buffer_id >= 1: │
│ 401 │ │ # import pdb; pdb.set_trace() │
│ ❱ 402 │ │ outputs = super(PipelineEngine, self).forward(inputs) │
│ 403 │ │ │
│ 404 │ │ # Reset activation checkpointing buffers. │
│ 405 │ │ # Need to call this between evaluation iterations │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/utils/nvtx.py:15 in wrapped_fn │
│ │
│ 12 │ │
│ 13 │ def wrapped_fn(*args, **kwargs): │
│ 14 │ │ get_accelerator().range_push(func.__qualname__) │
│ ❱ 15 │ │ ret_val = func(*args, **kwargs) │
│ 16 │ │ get_accelerator().range_pop() │
│ 17 │ │ return ret_val │
│ 18 │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/runtime/engine.py:1735 in forward │
│ │
│ 1732 │ │ if self.fp16_auto_cast(): │
│ 1733 │ │ │ inputs = self._cast_inputs_half(inputs) │
│ 1734 │ │ │
│ ❱ 1735 │ │ loss = self.module(*inputs, **kwargs) │
│ 1736 │ │ │
│ 1737 │ │ if self.zero_optimization_partition_weights(): │
│ 1738 │ │ │ # Disable automated discovery of external parameters │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/module.py:559 in forward │
│ │
│ 556 │ │ │ if "input_ids" in kwargs.keys() and isinstance(kwargs["input_ids"], dict): │
│ 557 │ │ │ │ return super(PipelineModel, self).forward(kwargs["input_ids"]) │
│ 558 │ │ │ else: │
│ ❱ 559 │ │ │ │ return super(PipelineModel, self).forward(*args, **kwargs) │
│ 560 │ │
│ 561 │ def get_input_embedding(self): │
│ 562 │ │ if env.pp_rank != 0: │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py:334 in │
│ forward │
│ │
│ 331 │ │ │
│ 332 │ │ if self.activation_checkpoint_interval == 0: │
│ 333 │ │ │ func = exec_range_func(0, len(self.forward_funcs)) │
│ ❱ 334 │ │ │ x = func(forward_input) │
│ 335 │ │ else: │
│ 336 │ │ │ num_layers = len(self.forward_funcs) │
│ 337 │ │ │ x = forward_input │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py:327 in │
│ exec_func │
│ │
│ 324 │ │ │ │ │ │ else: │
│ 325 │ │ │ │ │ │ │ ds_utils.set_random_seed(new_seed) │
│ 326 │ │ │ │ │ │
│ ❱ 327 │ │ │ │ │ inputs = layer(inputs) │
│ 328 │ │ │ │ return inputs │
│ 329 │ │ │ │
│ 330 │ │ │ return exec_func │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/models/llama/model.py:232 in forward │
│ │
│ 229 │ │
│ 230 │ def forward(self, inputs: dict): │
│ 231 │ │ if self.config.checkpointing and self.training: │
│ ❱ 232 │ │ │ inputs["hidden_states"] = torch.utils.checkpoint.checkpoint( │
│ 233 │ │ │ │ self._forward, │
│ 234 │ │ │ │ inputs["hidden_states"], │
│ 235 │ │ │ │ inputs.get("attention_mask", None) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/utils/checkpoint.py:249 in checkpoint │
│ │
│ 246 │ │ raise ValueError("Unexpected keyword arguments: " + ",".join(arg for arg in kwar │
│ 247 │ │
│ 248 │ if use_reentrant: │
│ ❱ 249 │ │ return CheckpointFunction.apply(function, preserve, *args) │
│ 250 │ else: │
│ 251 │ │ return _checkpoint_without_reentrant( │
│ 252 │ │ │ function, │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/autograd/function.py:506 in apply │
│ │
│ 503 │ │ if not torch._C._are_functorch_transforms_active(): │
│ 504 │ │ │ # See NOTE: [functorch vjp and autograd interaction] │
│ 505 │ │ │ args = _functorch.utils.unwrap_dead_wrappers(args) │
│ ❱ 506 │ │ │ return super().apply(*args, **kwargs) # type: ignore[misc] │
│ 507 │ │ │
│ 508 │ │ if cls.setup_context == _SingleLevelFunction.setup_context: │
│ 509 │ │ │ raise RuntimeError( │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/utils/checkpoint.py:107 in forward │
│ │
│ 104 │ │ ctx.save_for_backward(*tensor_inputs) │
│ 105 │ │ │
│ 106 │ │ with torch.no_grad(): │
│ ❱ 107 │ │ │ outputs = run_function(*args) │
│ 108 │ │ return outputs │
│ 109 │ │
│ 110 │ @staticmethod │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/models/llama/model.py:183 in │
│ _forward │
│ │
│ 180 │ │ │ start_pos = self.past_key_values.shape[3] │
│ 181 │ │ else: │
│ 182 │ │ │ start_pos = 0 │
│ ❱ 183 │ │ query, key = self.self_attn["rotary_emb"](query, key, seq_len, start_pos) │
│ 184 │ │ if self.past_key_values is not None: │
│ 185 │ │ │ query = torch.cat([self.past_key_values[0].permute([0, 2, 1, 3]), query], di │
│ 186 │ │ │ key = torch.cat([self.past_key_values[0].permute([0, 2, 1, 3]), key], dim=1) │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /d1/conda3/envs/scx_llm/lib/python3.10/site-packages/collie/models/llama/model.py:63 in forward │
│ │
│ 60 │ │ shape = [d if i == 1 or i == query.ndim - │
│ 61 │ │ │ │ 1 else 1 for i, d in enumerate(query.shape)] │
│ 62 │ │ freqs_cis = freqs_cis.view(*shape) │
│ ❱ 63 │ │ query = torch.view_as_real(query * freqs_cis).flatten(3) │
│ 64 │ │ key = torch.view_as_real(key * freqs_cis).flatten(3) │
│ 65 │ │ return query.type(t), key.type(t) │
│ 66 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cpu!把pp_size调成1才能避免报错，但是跑的时候loss一大半时间是nan。
用的dev分支，examples/further_pretrain_llama里的脚本，运行指令是
只修改了llama的路径，包括config、tokenizer和model.from_pretrained。报错如下：