怎么并行训练？ - Githubissues

我试图用 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 这样并行训练，会直接报错

ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 594501) of binary: /DaTa/mambaforge/bin/python3.10

完整log如下：

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:206 in         │
│ <module>                                                                                         │
│                                                                                                  │
│   203                                                                                            │
│   204 if __name__ == "__main__":                                                                 │
│   205 │   args = parse_args()                                                                    │
│ ❱ 206 │   train(args)                                                                            │
│   207                                                                                            │
│   208                                                                                            │
│                                                                                                  │
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:200 in train   │
│                                                                                                  │
│   197 │   │   data_collator=data_collator                                                        │
│   198 │   )                                                                                      │
│   199 │                                                                                          │
│ ❱ 200 │   trainer.train(resume_from_checkpoint=resume_from_checkpoint)                           │
│   201 │   trainer.model.save_pretrained(hf_train_args.output_dir)                                │
│   202                                                                                            │
│   203                                                                                            │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1645 in │
│ train                                                                                            │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1756 in │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   1753 │   │   │   │   if self.use_apex:                                                         │
│   1754 │   │   │   │   │   model = self.accelerator.prepare(self.model)                          │
│   1755 │   │   │   │   else:                                                                     │
│ ❱ 1756 │   │   │   │   │   model, self.optimizer = self.accelerator.prepare(self.model, self.op  │
│   1757 │   │   │   else:                                                                         │
│   1758 │   │   │   │   # to handle cases wherein we pass "DummyScheduler" such as when it is sp  │
│   1759 │   │   │   │   model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(      │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1182  │
│ in prepare                                                                                       │
│                                                                                                  │
│   1179 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│ ❱ 1182 │   │   │   result = tuple(                                                               │
│   1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1183  │
│ in <genexpr>                                                                                     │
│                                                                                                  │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│   1182 │   │   │   result = tuple(                                                               │
│ ❱ 1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│   1186                                                                                           │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1022  │
│ in _prepare_one                                                                                  │
│                                                                                                  │
│   1019 │   │   │   if isinstance(obj, torch.utils.data.DataLoader):                              │
│   1020 │   │   │   │   return self.prepare_data_loader(obj, device_placement=device_placement)   │
│   1021 │   │   │   elif isinstance(obj, torch.nn.Module):                                        │
│ ❱ 1022 │   │   │   │   return self.prepare_model(obj, device_placement=device_placement)         │
│   1023 │   │   │   elif isinstance(obj, torch.optim.Optimizer):                                  │
│   1024 │   │   │   │   optimizer = self.prepare_optimizer(obj, device_placement=device_placemen  │
│   1025 │   │   │   │   return optimizer                                                          │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1247  │
│ in prepare_model                                                                                 │
│                                                                                                  │
│   1244 │   │   ):                                                                                │
│   1245 │   │   │   model_devices = set(model.hf_device_map.values())                             │
│   1246 │   │   │   if len(model_devices) > 1 and self.distributed_type != DistributedType.NO:    │
│ ❱ 1247 │   │   │   │   raise ValueError(                                                         │
│   1248 │   │   │   │   │   "You can't train a model that has been loaded in 8-bit precision on   │
│   1249 │   │   │   │   │   " In order to use 8-bit models that have been loaded across multiple  │
│   1250 │   │   │   │   │   " Therefore you should not specify that you are under any distribute  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:206 in         │
│ <module>                                                                                         │
│                                                                                                  │
│   203                                                                                            │
│   204 if __name__ == "__main__":                                                                 │
│   205 │   args = parse_args()                                                                    │
│ ❱ 206 │   train(args)                                                                            │
│   207                                                                                            │
│   208                                                                                            │
│                                                                                                  │
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:200 in train   │
│                                                                                                  │
│   197 │   │   data_collator=data_collator                                                        │
│   198 │   )                                                                                      │
│   199 │                                                                                          │
│ ❱ 200 │   trainer.train(resume_from_checkpoint=resume_from_checkpoint)                           │
│   201 │   trainer.model.save_pretrained(hf_train_args.output_dir)                                │
│   202                                                                                            │
│   203                                                                                            │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1645 in │
│ train                                                                                            │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1756 in │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   1753 │   │   │   │   if self.use_apex:                                                         │
│   1754 │   │   │   │   │   model = self.accelerator.prepare(self.model)                          │
│   1755 │   │   │   │   else:                                                                     │
│ ❱ 1756 │   │   │   │   │   model, self.optimizer = self.accelerator.prepare(self.model, self.op  │
│   1757 │   │   │   else:                                                                         │
│   1758 │   │   │   │   # to handle cases wherein we pass "DummyScheduler" such as when it is sp  │
│   1759 │   │   │   │   model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(      │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1182  │
│ in prepare                                                                                       │
│                                                                                                  │
│   1179 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│ ❱ 1182 │   │   │   result = tuple(                                                               │
│   1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1183  │
│ in <genexpr>                                                                                     │
│                                                                                                  │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│   1182 │   │   │   result = tuple(                                                               │
│ ❱ 1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│   1186                                                                                           │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1022  │
│ in _prepare_one                                                                                  │
│                                                                                                  │
│   1019 │   │   │   if isinstance(obj, torch.utils.data.DataLoader):                              │
│   1020 │   │   │   │   return self.prepare_data_loader(obj, device_placement=device_placement)   │
│   1021 │   │   │   elif isinstance(obj, torch.nn.Module):                                        │
│ ❱ 1022 │   │   │   │   return self.prepare_model(obj, device_placement=device_placement)         │
│   1023 │   │   │   elif isinstance(obj, torch.optim.Optimizer):                                  │
│   1024 │   │   │   │   optimizer = self.prepare_optimizer(obj, device_placement=device_placemen  │
│   1025 │   │   │   │   return optimizer                                                          │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1247  │
│ in prepare_model                                                                                 │
│                                                                                                  │
│   1244 │   │   ):                                                                                │
│   1245 │   │   │   model_devices = set(model.hf_device_map.values())                             │
│   1246 │   │   │   if len(model_devices) > 1 and self.distributed_type != DistributedType.NO:    │
│ ❱ 1247 │   │   │   │   raise ValueError(                                                         │
│   1248 │   │   │   │   │   "You can't train a model that has been loaded in 8-bit precision on   │
│   1249 │   │   │   │   │   " In order to use 8-bit models that have been loaded across multiple  │
│   1250 │   │   │   │   │   " Therefore you should not specify that you are under any distribute  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 594501) of binary: /DaTa/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/DaTa/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346
  , in wrapper
    return f(*args, **kwargs)
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train_qlora.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-07-06_12:13:14
  host      : zhongshanyanke
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 594502)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-07-06_12:13:14
  host      : zhongshanyanke
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 594501)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

shuxueslpi / chatGLM-6B-QLoRA

怎么并行训练？ #18