Training issue on Multiple Gpu

DerrickXuNu commented 1 year ago

Dear authors,

First of all, thanks for your great work and the efforts to opensource this project! I met some issues while training the model. I am able to train on single GPU, but when I give --num_gpus 2 to train.py, it throws me this error:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Do you have any idea how to fix this? Thanks!

yaoshanliang commented 1 year ago

Same error

ejhung commented 1 year ago

I'm also working on reproducing the training step according to the author's manual. I tried to revise the issue of train.py last week, so not sure my fix would resolve your issue. What I've changed is:

# original
    if opt.num_gpus > 0:
        model = model.cuda()
# revised
    if opt.num_gpus > 0:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        _model = model.cuda()
        model = nn.DataParallel(_model).to(device)

Hope it helps you.

BTW, leveraging multi gpus with train.py script wasn't working for me since DDP functions are not included in the script. So in my case, I'm trying with train_ddp.py for training with multi gpus, but heading various issues as well.

happyday-lkj commented 1 year ago

I'm also working on reproducing the training step according to the author's manual. I tried to revise the issue of train.py last week, so not sure my fix would resolve your issue. What I've changed is:
# original
    if opt.num_gpus > 0:
        model = model.cuda()
# revised
    if opt.num_gpus > 0:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        _model = model.cuda()
        model = nn.DataParallel(_model).to(device)
Hope it helps you.

BTW, leveraging multi gpus with train.py script wasn't working for me since DDP functions are not included in the script. So in my case, I'm trying with train_ddp.py for training with multi gpus, but heading various issues as well.

Hello, I train the model with train_ddp.py for multi gpus, I meet some issues like https://github.com/datvuthanh/HybridNets/issues/90

datvuthanh / HybridNets

Training issue on Multiple Gpu #81