Closed happyday-lkj closed 1 year ago
when --num_gpus set to 1, the program work well, but when the gpu_nums set greater than 1, then the program seems stuck.
solved
@happyday-lkj hey, I've encountered the same problem. How did you fix that?
@happyday-lkj hey, I've encountered the same problem. How did you fix that?
first, you can modidy the code class ModelWithLoss in train_ddp.py with the “multilabel” instead of model.seg_mode, then refactor the train_ddp.py follow this https://github.com/rentainhe/pytorch-distributed-training/tree/master
BTW, it also have some problems with val_ddp.py.
@happyday-lkj hey, I've encountered the same problem. How did you fix that?
first, you can modidy the code class ModelWithLoss in train_ddp.py with the “multilabel” instead of model.seg_mode, then refactor the train_ddp.py follow this https://github.com/rentainhe/pytorch-distributed-training/tree/master
BTW, it also have some problems with val_ddp.py.
thanks for your advice, I can train with DDP now.
Hi, I try train the model with DDP model, with script
python train_ddp.py --num_gpus 2
however, I meet the errorFile "HybridNets/hybridnets/loss.py", line 500, in soft_tversky_score assert output.size() == target.size() AssertionError
Then, I modidy the code class ModelWithLoss in train_ddp.py with the “multilabel” instead ofmodel.seg_mode
, but the code seems stopped .