Loading Weights After Training

[nirajkark07]

Hello,

I am trying to load my weights for inference after training. But I am running into the following error. What could be causing this issue, and how could I fix it? Thank you. :)

python inference.py --weights ../weights/net_epoch_60.pth --data ../gearbox_test --object Gearbox_Scaled Found 1 weights. Loading DOPE model '../weights/net_epoch_60.pth'... /home/niraj/output/dope_training/lib/python3.12/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /home/niraj/output/dope_training/lib/python3.12/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum orNonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=None. warnings.warn(msg) /home/niraj/Deep_Object_Pose/inference/../common/detector.py:269: FutureWarning: You are usingtorch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. net.load_state_dict(torch.load(path)) Traceback (most recent call last): File "/home/niraj/Deep_Object_Pose/inference/inference.py", line 258, in <module> dope_node = DopeNode(config, weight, opt.parallel, opt.object) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/niraj/Deep_Object_Pose/inference/inference.py", line 49, in __init__ self.model.load_net_model() File "/home/niraj/Deep_Object_Pose/inference/../common/detector.py", line 253, in load_net_model self.net = self.load_net_model_path(self.net_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/niraj/Deep_Object_Pose/inference/../common/detector.py", line 269, in load_net_model_path net.load_state_dict(torch.load(path)) File "/home/niraj/output/dope_training/lib/python3.12/site-packages/torch/nn/modules/module.py", line 2215, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for DopeNetwork: Missing key(s) in state_dict: "vgg.0.weight", "vgg.0.bias", "vgg.2.weight", "vgg.2.bias", "vgg.5.weight", "vgg.5.bias", "vgg.7.weight", "vgg.7.bias", "vgg.10.weight", "vgg.10.bias", "vgg.12.weight", "vgg.12.bias", "vgg.14.weight", "vgg.14.bias", "vgg.16.weight", "vgg.16.bias", "vgg.19.weight", "vgg.19.bias", "vgg.21.weight", "vgg.21.bias", "vgg.23.weight", "vgg.23.bias", "vgg.25.weight", "vgg.25.bias", "m1_2.0.weight", "m1_2.0.bias", "m1_2.2.weight", "m1_2.2.bias", "m1_2.4.weight", "m1_2.4.bias", "m1_2.6.weight", "m1_2.6.bias", "m1_2.8.weight", "m1_2.8.bias", "m2_2.0.weight", "m2_2.0.bias", "m2_2.2.weight", "m2_2.2.bias", "m2_2.4.weight", "m2_2.4.bias", "m2_2.6.weight", "m2_2.6.bias", "m2_2.8.weight", "m2_2.8.bias", "m2_2.10.weight", "m2_2.10.bias", "m2_2.12.weight", "m2_2.12.bias", "m3_2.0.weight", "m3_2.0.bias", "m3_2.2.weight", "m3_2.2.bias", "m3_2.4.weight", "m3_2.4.bias", "m3_2.6.weight", "m3_2.6.bias", "m3_2.8.weight", "m3_2.8.bias", "m3_2.10.weight", "m3_2.10.bias", "m3_2.12.weight", "m3_2.12.bias", "m4_2.0.weight", "m4_2.0.bias", "m4_2.2.weight", "m4_2.2.bias", "m4_2.4.weight", "m4_2.4.bias", "m4_2.6.weight", "m4_2.6.bias", "m4_2.8.weight", "m4_2.8.bias", "m4_2.10.weight", "m4_2.10.bias", "m4_2.12.weight", "m4_2.12.bias", "m5_2.0.weight", "m5_2.0.bias", "m5_2.2.weight", "m5_2.2.bias", "m5_2.4.weight", "m5_2.4.bias", "m5_2.6.weight", "m5_2.6.bias", "m5_2.8.weight", "m5_2.8.bias", "m5_2.10.weight", "m5_2.10.bias", "m5_2.12.weight", "m5_2.12.bias", "m6_2.0.weight", "m6_2.0.bias", "m6_2.2.weight", "m6_2.2.bias", "m6_2.4.weight", "m6_2.4.bias", "m6_2.6.weight", "m6_2.6.bias", "m6_2.8.weight", "m6_2.8.bias", "m6_2.10.weight", "m6_2.10.bias", "m6_2.12.weight", "m6_2.12.bias", "m1_1.0.weight", "m1_1.0.bias", "m1_1.2.weight", "m1_1.2.bias", "m1_1.4.weight", "m1_1.4.bias", "m1_1.6.weight", "m1_1.6.bias", "m1_1.8.weight", "m1_1.8.bias", "m2_1.0.weight", "m2_1.0.bias", "m2_1.2.weight", "m2_1.2.bias", "m2_1.4.weight", "m2_1.4.bias", "m2_1.6.weight", "m2_1.6.bias", "m2_1.8.weight", "m2_1.8.bias", "m2_1.10.weight", "m2_1.10.bias", "m2_1.12.weight", "m2_1.12.bias", "m3_1.0.weight", "m3_1.0.bias", "m3_1.2.weight", "m3_1.2.bias", "m3_1.4.weight", "m3_1.4.bias", "m3_1.6.weight", "m3_1.6.bias", "m3_1.8.weight", "m3_1.8.bias", "m3_1.10.weight", "m3_1.10.bias", "m3_1.12.weight", "m3_1.12.bias", "m4_1.0.weight", "m4_1.0.bias", "m4_1.2.weight", "m4_1.2.bias", "m4_1.4.weight", "m4_1.4.bias", "m4_1.6.weight", "m4_1.6.bias", "m4_1.8.weight", "m4_1.8.bias", "m4_1.10.weight", "m4_1.10.bias", "m4_1.12.weight", "m4_1.12.bias", "m5_1.0.weight", "m5_1.0.bias", "m5_1.2.weight", "m5_1.2.bias", "m5_1.4.weight", "m5_1.4.bias", "m5_1.6.weight", "m5_1.6.bias", "m5_1.8.weight", "m5_1.8.bias", "m5_1.10.weight", "m5_1.10.bias", "m5_1.12.weight", "m5_1.12.bias", "m6_1.0.weight", "m6_1.0.bias", "m6_1.2.weight", "m6_1.2.bias", "m6_1.4.weight", "m6_1.4.bias", "m6_1.6.weight", "m6_1.6.bias", "m6_1.8.weight", "m6_1.8.bias", "m6_1.10.weight", "m6_1.10.bias", "m6_1.12.weight", "m6_1.12.bias". Unexpected key(s) in state_dict: "module.vgg.0.weight", "module.vgg.0.bias", "module.vgg.2.weight", "module.vgg.2.bias", "module.vgg.5.weight", "module.vgg.5.bias", "module.vgg.7.weight", "module.vgg.7.bias", "module.vgg.10.weight", "module.vgg.10.bias", "module.vgg.12.weight", "module.vgg.12.bias", "module.vgg.14.weight", "module.vgg.14.bias", "module.vgg.16.weight", "module.vgg.16.bias", "module.vgg.19.weight", "module.vgg.19.bias", "module.vgg.21.weight", "module.vgg.21.bias", "module.vgg.23.weight", "module.vgg.23.bias", "module.vgg.25.weight", "module.vgg.25.bias", "module.m1_2.0.weight", "module.m1_2.0.bias", "module.m1_2.2.weight", "module.m1_2.2.bias", "module.m1_2.4.weight", "module.m1_2.4.bias", "module.m1_2.6.weight", "module.m1_2.6.bias", "module.m1_2.8.weight", "module.m1_2.8.bias", "module.m2_2.0.weight", "module.m2_2.0.bias", "module.m2_2.2.weight", "module.m2_2.2.bias", "module.m2_2.4.weight", "module.m2_2.4.bias", "module.m2_2.6.weight", "module.m2_2.6.bias", "module.m2_2.8.weight", "module.m2_2.8.bias", "module.m2_2.10.weight", "module.m2_2.10.bias", "module.m2_2.12.weight", "module.m2_2.12.bias", "module.m3_2.0.weight", "module.m3_2.0.bias", "module.m3_2.2.weight", "module.m3_2.2.bias", "module.m3_2.4.weight", "module.m3_2.4.bias", "module.m3_2.6.weight", "module.m3_2.6.bias", "module.m3_2.8.weight", "module.m3_2.8.bias", "module.m3_2.10.weight", "module.m3_2.10.bias", "module.m3_2.12.weight", "module.m3_2.12.bias", "module.m4_2.0.weight", "module.m4_2.0.bias", "module.m4_2.2.weight", "module.m4_2.2.bias", "module.m4_2.4.weight", "module.m4_2.4.bias", "module.m4_2.6.weight", "module.m4_2.6.bias", "module.m4_2.8.weight", "module.m4_2.8.bias", "module.m4_2.10.weight", "module.m4_2.10.bias", "module.m4_2.12.weight", "module.m4_2.12.bias", "module.m5_2.0.weight", "module.m5_2.0.bias", "module.m5_2.2.weight", "module.m5_2.2.bias", "module.m5_2.4.weight", "module.m5_2.4.bias", "module.m5_2.6.weight", "module.m5_2.6.bias", "module.m5_2.8.weight", "module.m5_2.8.bias", "module.m5_2.10.weight", "module.m5_2.10.bias", "module.m5_2.12.weight", "module.m5_2.12.bias", "module.m6_2.0.weight", "module.m6_2.0.bias", "module.m6_2.2.weight", "module.m6_2.2.bias", "module.m6_2.4.weight", "module.m6_2.4.bias", "module.m6_2.6.weight", "module.m6_2.6.bias", "module.m6_2.8.weight", "module.m6_2.8.bias", "module.m6_2.10.weight", "module.m6_2.10.bias", "module.m6_2.12.weight", "module.m6_2.12.bias", "module.m1_1.0.weight", "module.m1_1.0.bias", "module.m1_1.2.weight", "module.m1_1.2.bias", "module.m1_1.4.weight", "module.m1_1.4.bias", "module.m1_1.6.weight", "module.m1_1.6.bias", "module.m1_1.8.weight", "module.m1_1.8.bias", "module.m2_1.0.weight", "module.m2_1.0.bias", "module.m2_1.2.weight", "module.m2_1.2.bias", "module.m2_1.4.weight", "module.m2_1.4.bias", "module.m2_1.6.weight", "module.m2_1.6.bias", "module.m2_1.8.weight", "module.m2_1.8.bias", "module.m2_1.10.weight", "module.m2_1.10.bias", "module.m2_1.12.weight", "module.m2_1.12.bias", "module.m3_1.0.weight", "module.m3_1.0.bias", "module.m3_1.2.weight", "module.m3_1.2.bias", "module.m3_1.4.weight", "module.m3_1.4.bias", "module.m3_1.6.weight", "module.m3_1.6.bias", "module.m3_1.8.weight", "module.m3_1.8.bias", "module.m3_1.10.weight", "module.m3_1.10.bias", "module.m3_1.12.weight", "module.m3_1.12.bias", "module.m4_1.0.weight", "module.m4_1.0.bias", "module.m4_1.2.weight", "module.m4_1.2.bias", "module.m4_1.4.weight", "module.m4_1.4.bias", "module.m4_1.6.weight", "module.m4_1.6.bias", "module.m4_1.8.weight", "module.m4_1.8.bias", "module.m4_1.10.weight", "module.m4_1.10.bias", "module.m4_1.12.weight", "module.m4_1.12.bias", "module.m5_1.0.weight", "module.m5_1.0.bias", "module.m5_1.2.weight", "module.m5_1.2.bias", "module.m5_1.4.weight", "module.m5_1.4.bias", "module.m5_1.6.weight", "module.m5_1.6.bias", "module.m5_1.8.weight", "module.m5_1.8.bias", "module.m5_1.10.weight", "module.m5_1.10.bias", "module.m5_1.12.weight", "module.m5_1.12.bias", "module.m6_1.0.weight", "module.m6_1.0.bias", "module.m6_1.2.weight", "module.m6_1.2.bias", "module.m6_1.4.weight", "module.m6_1.4.bias", "module.m6_1.6.weight", "module.m6_1.6.bias", "module.m6_1.8.weight", "module.m6_1.8.bias", "module.m6_1.10.weight", "module.m6_1.10.bias", "module.m6_1.12.weight", "module.m6_1.12.bias".

[nirajkark07]

Resolved using the advice provided in [https://github.com/NVlabs/Deep_Object_Pose/issues/25(url).

[zhaowangji]

for those who does not understand at once:

from the problem report: File "../common/detector.py", line 253, in load_net_model self.net = self.load_net_model_path(self.net_path) File "../common/detector.py", line 269, in load_net_model_path net.load_state_dict(torch.load(path)) File "/home/lvpin/.conda/envs/torch1.12.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for DopeNetwork: Missing key(s) in state_dict: "vgg.0.weight", " ……………………

we go to /common/detector.py and find "load_state_dict"

previous code:

    net.load_state_dict(torch.load(path))

now code:(just copy from the post which is mentioned and a little modification)

    # original saved file with DataParallel
    state_dict = torch.load(path)
    # create new OrderedDict that does not contain `module.`
    from collections import OrderedDict
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = k[7:] # remove `module.`
        new_state_dict[name] = v
    # load params
    net.load_state_dict(new_state_dict)

then everything is fine again!