Is Torchmeta compatible with Dataparallel?

[smounsav]

Hi, is torchmeta compatible with dataparallel? I have troubles training a MetaModule wrapped by dataparallel. It seems get_subdict has problems with the layer names starting with "module." Thanks for your help!

[mahf93]

I also have the same problem, would appreciate an answer.

[tristandeleu]

Hi, sorry for the late reply. I have had similar other feedbacks regarding compatibility between MetaModules and Dataparallel. I will look into it.

[tristandeleu]

Can you provide a minimal example of this issue?

[Qunxiang]

Can you provide a minimal example of this issue?

Hi, when I use multi gpus to train the model like this,

model = torch.nn.DataParallel(model)

it raises AttributeError: 'DataParallel' object has no attribute 'meta_parameters'

Thanks for your help!

[tristandeleu]

Thank you! I will test this, but it looks like this might require an extension of DataParallel in Torchmeta to handle the meta_parameters attribute.

[tristandeleu]

I have added support for DataParallel. Unfortunately it does not work out of the box with torch.nn.DataParallel, but I have added DataParallel in torchmeta.modules which is identical to the torch version, with full support for MetaModule instances (pass a params argument in forward(), access meta_parameters()). This means that you probably would have to update your code replacing torch.nn.DataParallel with torchmeta.modules.DataParallel. Here is a basic example:

import torch
import torch.nn as nn

from torchmeta.modules import MetaSequential, MetaLinear
from torchmeta.modules import DataParallel

model = MetaSequential(
    MetaLinear(2, 3),
    nn.ReLU(),
    MetaLinear(3, 1))
model = DataParallel(model) # Instead of model = nn.DataParallel(model)
model.to(device=torch.device('cuda:0'))

Please let me know if that works for you, and if you find any bugs. I am not usually using multi-GPU, so I won't be able to fully test it myself.

[smounsav]

Thanks a lot! I'll try it and let you know I find any bug.