search

hpcaitech / ColossalAI-Examples

Examples of training models with hybrid parallelism using ColossalAI
Apache License 2.0
333 stars 102 forks source link

grad is none when run gpt2 with pipeline parallelism only #204

Open lin88lin8850 opened 1 year ago

lin88lin8850 commented 1 year ago

🐛 Describe the bug

image

Environment

torch==1.12.0a0+8a1a93a num_gpu=4 pipeline=4 model = gpt2-small