Problem when try to run inference

[zhan-xu]

Hi all, I am trying to run simple inference of this code. As the author said, two ways are:

(1) python tools/test_net.py --dataset coco2017 --cfg config/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml --load_ckpt {path/to/your/checkpoint}

(2) python tools/infer_simple.py --dataset coco --cfg cfgs/baselines/e2e_mask_rcnn_R-50-C4.yml --load_ckpt {path/to/your/checkpoint} --image_dir {dir/of/input/images} --output_dir {dir/to/save/visualizations}

For (1), the checkpoint is resnet50_caffe.pth, right? When I ran --load_ckpt data/pretrained_model/resnet50_caffe.pth, I got the error

"net_utils.load_ckpt(model, checkpoint['model'])   KeyError: 'model'"

I also tried "--load_detectron data/pretrained_model/R-50.pkl" and changed the "IMAGENET_PRTETRAINED_WEIGHTS" in "e2e_mask_rcnn_R-50-C4_1x.yaml" to "data/pretrained_model/R-50.pkl", but get the error:

 "File "/mnt/gypsum/home/zhanxu/Proj/mask-rcnn.pytorch/lib/utils/detectron_weight_helper.py", line 21, in load_detectron_weight
  p_tensor.copy_(torch.Tensor(src_blobs[d_name]))
  KeyError: 'fpn_inner_res5_2_sum_w'"

For (2), there is no "e2e_mask_rcnn_R-50-C4.yml". Is it e2e_mask_rcnn_R-50-C4_1x.yaml or e2e_mask_rcnn_R-50-C4_2x.yaml?

Can someone help me with a suitable command that can run inference?

[YanShuo1992]

I meet the same problem. I believe the provided in repo is only the weights of classification. You need to train the it before evaluating it.

If I am wrong, some one please correct me.

[vsd550]

Yes u need to understand that --load_detectron takes a pretrained model. U can find it at model zoo in the link https://github.com/facebookresearch/Detectron/blob/master/MODEL_ZOO.md

Try downloading the model file, say R-50-C4 from the section End-to-End Faster & Mask R-CNN Baselines and put it in another directory, say detectron_weights and provide that path. Hope it works

[triangleCZH]

I have a relevant question. I'm not sure what's the difference between .pth and .pkl weights and what exactly they are used for? when I try to run inference with loading them, none of them contains the dictionary structure with 'step' and 'model', but when I train a model for some steps, I find .pth containing 'model' 'step' keys, and .pkl files are not used.

[JoshuaLelon]

@triangleCZH

I've ran a lot of different combinations of training attempts, and something I've noticed is that .pth checkpoints never work with --load_detectron and .pkl checkpoints never work with --load_ckpt. I'm not sure why though.

[triangleCZH]

@joshualmitchell

Thank you for replying. I feel the same. Do you know if there is the difference between --load_detectron and --load_ckpt?

[lsrock1]

@triangleCZH --load_detectron is for loading pretrained detectron in original caffe model(pkl or pth) but in this case original caffe model's weight name(layer name) is different with pytorch version's layer name. so to load model to torch tensor, detectron uses some function that changes layer name

--load_ckpt is for loading pretrained model that you have trained in this case everything is fine. just load

That is why ckpt and detectron were not working

pkl file is a kind of dictionary file (weigth name: weight)that is saved by python pickle library. pth file is binary model file that is saved by torch.save pkl and pth files are handled different way.

[triangleCZH]

@lsrock1 Thank you for your kind and detailed explanation. That helps a lot!

[wangfei-cs]

Have you fixed it? I met a same problem @enderhsu @YanShuo1992

python tools/infer_simple.py --dataset coco --cfg configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml --load_ckpt data/pretrained_model/resnet50_caffe.pth --images ~/image/farm/img1_keypoints-pydetectron-R50-FPN.jpg --output_dir ~/image/farm_out Called with args: Namespace(cfg_file='configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml', cuda=True, dataset='coco', image_dir=None, images=['/home/feiw/image/farm/img1_keypoints-pydetectron-R50-FPN.jpg'], load_ckpt='data/pretrained_model/resnet50_caffe.pth', load_detectron=None, merge_pdfs=True, output_dir='/home/feiw/image/farm_out', set_cfgs=[]) load cfg from file: configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml loading checkpoint data/pretrained_model/resnet50_caffe.pth Traceback (most recent call last): File "tools/infer_simple.py", line 176, in main() File "tools/infer_simple.py", line 124, in main net_utils.load_ckpt(maskRCNN, checkpoint['model']) KeyError: 'model'

[wangfei-cs]

Also,

python tools/infer_simple.py --dataset coco --cfg configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml --load_detectron data/pretrained_model/R-50.pkl --images ~/image/farm/img1_keypoints-pydetectron-R50-FPN.jpg --output_dir ~/image/farm_out Called with args: Namespace(cfg_file='configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml', cuda=True, dataset='coco', image_dir=None, images=['/home/feiw/image/farm/img1_keypoints-pydetectron-R50-FPN.jpg'], load_ckpt=None, load_detectron='data/pretrained_model/R-50.pkl', merge_pdfs=True, output_dir='/home/feiw/image/farm_out', set_cfgs=[]) load cfg from file: configs/baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml loading detectron weights data/pretrained_model/R-50.pkl Traceback (most recent call last): File "tools/infer_simple.py", line 176, in main() File "tools/infer_simple.py", line 128, in main load_detectron_weight(maskRCNN, args.load_detectron) File "/home/feiw/Detectron/Detectron.pytorch/lib/utils/detectron_weight_helper.py", line 21, in load_detectron_weight ptensor.copy(torch.Tensor(src_blobs[d_name])) KeyError: 'fpn_inner_res5_2_sum_w'

[baiyancheng20]

I also met with this problem. I checked the code 15 src_blobs = src_blobs['blobs'], there is no key 'fpn_inner_res5_2_sum_w' in the dict src_blobs.

[sararojas]

@wangfei-cs are you sure that you are using --dataset keypoints_coco2017? because I had the same issue and then, I solved with that! :)

[jamesjjcondon]

I have the same two problems as @enderhsu - --load_ckpt data/pretrained_model/resnet50_caffe.pth throws

"net_utils.load_ckpt(model, checkpoint['model']) KeyError: 'model'"

and --load_detectron data/pretrained_model/R-50.pkl with "IMAGENET_PRTETRAINED_WEIGHTS" in "e2e_mask_rcnn_R-50-C4_1x.yaml" changed to "data/pretrained_model/R-50.pkl" throws

 "File "/mnt/gypsum/home/zhanxu/Proj/mask-rcnn.pytorch/lib/utils/detectron_weight_helper.py", line 21, in load_detectron_weight
  p_tensor.copy_(torch.Tensor(src_blobs[d_name]))
  KeyError: 'fpn_inner_res5_2_sum_w'"

Having trouble trying to inspect the .pkl and .pth files to try and solve. Thanks @triangleCZH !

But has anyone found a work-around?

[jamesjjcondon]

Got testing with --cfg configs/baselines/e2e_faster_rcnn_R-50-C4_1x.yaml --load_detectron data/pretrained_model/model_final.pkl from https://github.com/facebookresearch/Detectron/blob/master/MODEL_ZOO.md thanks @vsd550 . Plus ``` 'RESNETS': {'FREEZE_AT': 2, 'IMAGENET_PRETRAINED_WEIGHTS': 'data/pretrained_model/resnet50_caffe.pth'

Had to adjust for memory with my machine in .yaml at: TEST: ... MAX_SIZE: _____

[andybarry]

For other folks, I also solved this by following @jamesjjcondon above, downloading this model:

End-to-End Faster & Mask R-CNN Baselines
R-50-C4 | Mask | 2x | 1 | 6.6 | 0.620 | 62.0 | 0.182 + 0.017 | 37.8 | 32.8 | - | - | 35858828

(model ID: 35858828)

Working line for me:

python tools/infer_simple.py --cfg configs/baselines/e2e_faster_rcnn_R-50-C4_1x.yaml --load_detectron=data/pretrained_model/model_final.pkl --image_dir imgs_in  --output_dir imgs_out --dataset coco

[Merofine]

Thanks i work.!