JINGTING92
commented
1 year ago same problem, any solutions?
frame #56: __libc_start_main + 0xe7 (0x7f5d250ddc87 in /lib/x86_64-linux-gnu/libc.so.6) frame #57: _start + 0x2a (0x55a1f2d1f4ca in /usr/bin/python)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f09f2bd94d7 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::string const&) + 0x64 (0x7f09f2ba336b in /usr/local/lib/python3.8/site-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const, char const, int, bool) + 0x118 (0x7f09f2c7dfa8 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)**
rSemology
molly-smith
LeeChongKeat
johnchienbronci
Hprairie
I trained the PPO model, use the gpt. I modified the option of model_name_or_path from opt to gpt2 I passed step 1 and step 2,But An error occurred in step 3.The error is as follows: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /data/luojiangang/423_Deep/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_fin │ │ etuning/main.py:522 in │
│ │
│ 519 │
│ 520 │
│ 521 if name == "main": │
│ ❱ 522 │ main() │
│ 523 │
│ │
│ /data/luojiangang/423_Deep/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_fin │
│ etuning/main.py:431 in main │
│ │
│ 428 │ │ │ │ prompts = prompts[:, length - args.max_prompt_seq_len:] │
│ 429 │ │ │ │ raise ValueError("Prompt length is too long") │
│ 430 │ │ │ │
│ ❱ 431 │ │ │ out = trainer.generate_experience(prompts) │
│ 432 │ │ │ exp_dataset = exp_mini_dataset.add(out) │
│ 433 │ │ │ │
│ 434 │ │ │ if exp_dataset is not None: │
│ │
│ /data/luojiangang/423_Deep/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_fin │
│ etuning/ppo_trainer.py:97 in generate_experience │
│ │
│ 94 │ │
│ 95 │ def generate_experience(self, prompts): │
│ 96 │ │ self.eval() │
│ ❱ 97 │ │ seq = self._generate_sequence(prompts) │
│ 98 │ │ self.train() │
│ 99 │ │ │
│ 100 │ │ pad_token_id = self.tokenizer.pad_token_id │
│ │
│ /data/luojiangang/423_Deep/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_fin │
│ etuning/ppo_trainer.py:91 in _generate_sequence │
│ │
│ 88 │ │ │ │ continue │
│ 89 │ │ │ else: │
│ 90 │ │ │ │ out_seq.append(seq[i:i + 1]) │
│ ❱ 91 │ │ out_seq = torch.cat(out_seq, dim=0) # concate output in the batch dim │ │ 92 │ │ │ │ 93 │ │ return out_seq │ │ 94 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: torch.cat(): expected a non-empty list of Tensors
torch.Size([4, 50264])
torch.Size([4, 50264])
!!!! kernel execution error. (m: 2048, n: 4, k: 2048, error: 14)
!!!! kernel execution error. (m: 8192, n: 4, k: 2048, error: 13)
!!!! kernel execution error. (m: 2048, n: 4, k: 2048, error: 13)
Do you know what causes this? Can you provide the training steps for gpt2.Looking forward to your reply