How to extract ResNet-50 from open-mmlab://detectron2/resnet50_caffe and pre-train it?

[sarmientoj24]

Already saw how to use a custom dataset. I was just wondering if I ever need to extract ResNet-50 from that or is it already resnet-50 and I can already use it for pre-training?

[iseessel]

Hi @sarmientoj24 -- thanks for reaching out! I am not sure I entirely understand the question -- can you elaborate on what your goal is?

To use detectron2 model weights, you'll have to convert them to be VISSL compatible: We have a script for this -- \see https://vissl.readthedocs.io/en/v0.1.5/compatibility_libraries.html?highlight=detectron2#converting-caffe2-models-vissl.

To use a resnet-50 architecture, see for example this config: https://github.com/facebookresearch/vissl/blob/main/configs/config/pretrain/supervised/supervised_8gpu_resnet.yaml -- which trains a model supervised on imagenet. Specifically we have the following entries in our config:

  MODEL:
    TRUNK:
      NAME: resnet
      RESNETS:
        DEPTH: 50

[sarmientoj24]

@iseessel Thank you for the response! Appreciate it. Great library!

My goal is to use the pre-trained weights from mmdetection (ResNet-50 Caffe model, which I am using for Semi-Supervised Learning). Below is the config from mmdetection. But it seems like it is from Detectron2 as well.

model = dict(
    backbone=dict(
        norm_eval=True,
        style="caffe",
        init_cfg=dict(
            type="Pretrained", checkpoint="open-mmlab://detectron2/resnet50_caffe"
        ),
    )
)

This guide looks pretty neat:

https://vissl.readthedocs.io/en/v0.1.5/compatibility_libraries.html?highlight=detectron2#converting-caffe2-models-vissl.

Although I was wondering how do I put it back as Caffe since I shall be using it for other task that requires it to be Caffe? Is it this? And just change the --weights_type torch to caffe?

python extra_scripts/convert_vissl_to_detectron2.py \
    --input_model_file <input_model>.pth  \
    --output_model <d2_model>.torch \
    --weights_type torch \
    --state_dict_key_name classy_state_dict

Second question, if I use the Pytorch ResNet from Detectron2, do I still need to convert it to VISSL?

[iseessel]

_Although I was wondering how do I put it back as Caffe since I shall be using it for other task that requires it to be Caffe? Is it this? And just change the --weightstype torch to caffe?

@sarmientoj24 Yep that looks correct to me!

Second question, if I use the Pytorch ResNet from Detectron2, do I still need to convert it to VISSL?

You should be able to load it directly, assuming it is a torchvision model. See this tutorial: https://vissl.ai/tutorials/Benchmark_Linear_Image_Classification_on_ImageNet_1K_V0_1_6 where we load a torchvision model.

We offer pretty direct logs that will tell you which model weights were loaded. See the above tutorial and specifically this part of the training logs:

INFO 2021-10-18 00:37:02,228 util.py: 276: Attempting to load checkpoint from /content/resnet50-19c8e357.pth
INFO 2021-10-18 00:37:02,440 util.py: 281: Loaded checkpoint from /content/resnet50-19c8e357.pth
INFO 2021-10-18 00:37:02,440 util.py: 240: Broadcasting checkpoint loaded from /content/resnet50-19c8e357.pth
INFO 2021-10-18 00:37:06,165 train_task.py: 429: Checkpoint loaded: /content/resnet50-19c8e357.pth...
...
INFO 2021-10-18 00:37:06,327 checkpoint.py: 851: Ignored layer: trunk._feature_blocks.layer4.2.bn3.num_batches_tracked
INFO 2021-10-18 00:37:06,327 checkpoint.py: 894: Not found:     heads.0.channel_bn.weight, not initialized
INFO 2021-10-18 00:37:06,327 checkpoint.py: 894: Not found:     heads.0.channel_bn.bias, not initialized
INFO 2021-10-18 00:37:06,327 checkpoint.py: 894: Not found:     heads.0.channel_bn.running_mean, not initialized
INFO 2021-10-18 00:37:06,327 checkpoint.py: 894: Not found:     heads.0.channel_bn.running_var, not initialized
INFO 2021-10-18 00:37:06,327 checkpoint.py: 851: Ignored layer: heads.0.channel_bn.num_batches_tracked
INFO 2021-10-18 00:37:06,328 checkpoint.py: 894: Not found:     heads.0.clf.clf.0.weight, not initialized
INFO 2021-10-18 00:37:06,328 checkpoint.py: 894: Not found:     heads.0.clf.clf.0.bias, not initialized
INFO 2021-10-18 00:37:06,328 checkpoint.py: 901: Extra layers not loaded from checkpoint: ['trunk._feature_blocks.fc.weight', 'trunk._feature_blocks.fc.bias', 'trunk._feature_blocks.type']

You can see here we loaded the entire trunk and ommitted the Head. You can look at these logs (log.txt) to verify your weights were loaded correctly.

[sarmientoj24]

@iseessel Seems like convert_caffe2_to_torchvision_resnet.py can only take .pkl files and not .pth. I downloaded the Caffe2 Detectron model (.pth) from here and it doesn't work.

There is a .pkl weights file for that mmlab detectron2 caffe model here https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-50.pkl

This one works with your converter. However, after using ViSSL, how do I convert it to mmlab's detectron2 caffe2 .pth?

[iseessel]

Glad the .pkl worked.

To convert back to detectron2, you can follow this: https://vissl.readthedocs.io/en/v0.1.6/evaluations/object_detection.html?highlight=object%20detection#benchmark-task-object-detection