how to convert pretrianed caffe2 R2Plus1D models to pytorch?

[cdeepakroy]

How can i convert the pretrained caffe2 R2Plus1D models listed [here] (https://github.com/facebookresearch/R2Plus1D/blob/master/tutorials/models.md) into a pytorch model for fine-tuning on a custom dataset? I will be happy to contribute the models back once i succeed.

[XiongChengxin]

Could it be possible to according to the architecture of pytorch model and caffe2 model, adjust the parameters of each layer and save?

[dutran]

@cdeepakroy yes, you can because models now are just sets of pickle blobs.

[XiongChengxin]

@cdeepakroy Hi, Have you successed in converting caffe2 model into pytorch?

[dontLoveBugs]

How can i convert the pretrained caffe2 R2Plus1D models listed [here] (https://github.com/facebookresearch/R2Plus1D/blob/master/tutorials/models.md) into a pytorch model for fine-tuning on a custom dataset? I will be happy to contribute the models back once i succeed.

Hi, I also want to know how to convert the pretrained caffe2 R2Plus1D models. Have you successed？

[dutran]

We are planning to release more models in both caffe2 and pytorch (pytorch models may be later due to missing Video loader).

[bjuncek]

Converting the models to pytorch is relatively straightforward but painful process (needs to be done by hand). You can find the sample model definitions here and you'd need to copy all the weights by hand.

Cheers

[inspiros]

@bjuncek Thank you for your converted models Bruno!

But is that just really just copying from the .pkl file and assigning to pytorch's statedict? I don't see the weights of your converted r2plus1d_18-91a641e6.pth equal to weights of r2.5d_d18_l16.pkl from pre-trained models and somehow have 2 millions less Params.

Am I missing something? Please tell me I'm trying to convert the batch-norm C3D model and it's giving me false label.

[bjuncek]

So, you'd need to assign the correct weight to correct layer. For R3D, that would look something like:

def get_blob(weights, name):
    return torch.from_numpy(weights['blobs'][name])

model.conv1.weight.data = get_blob(weights, "conv1_w")
model.bn1.weight.data = get_blob(weights, 'conv1_spatbn_relu_s')
model.bn1.running_mean.data = get_blob(weights, 'conv1_spatbn_relu_rm')
model.bn1.running_var.data = get_blob(weights,  'conv1_spatbn_relu_riv')
model.bn1.bias.data = get_blob(weights, 'conv1_spatbn_relu_b')

for i in range(0,2):
    layer_counter = 0
    model.layer1[i % 2].conv1.weight.data = get_blob(weights,'comp_{}_conv_1_w'.format(layer_counter+i))
    model.layer1[i % 2].bn1.weight.data = get_blob(weights, 'comp_{}_spatbn_1_s'.format(layer_counter+i))
    model.layer1[i % 2].bn1.bias.data = get_blob(weights, 'comp_{}_spatbn_1_b'.format(layer_counter+i))

...
...

The pre-trained models referenced in the PR (and subsequent refactoring which makes them more readable and accurate) were actually trained from scratch to match the performance of the models here, so the weights wouldn't be the same.

[daniel-j-h]

Hey folks we just did that for the r(2+1)d 34-layer model here:

https://github.com/moabitcoin/ig65m-pytorch

Hope that helps.

[bjuncek]

Thanks @daniel-j-h. Much appreciated!

[bjuncek]

ps, @cdeepakroy if you found this solution satisfactory, feel free to close the issue. Many thanks!!!

[dutran]

merge to https://github.com/facebookresearch/VMZ/issues/86