how to make faster-rcnn use the coco caffemodel and not the VGG16 caffemodel?

[monajalal]

Goal: Use faster-RCNN to detect mscoco classes in random objects off the web. I am running this code (slightly modified version of tools/demo.py) and I am not sure what I should change in which setting files so that it would use the mscoco .caffemodel? https://gist.github.com/monajalal/07f891480a49c7559099d5604d5340ad When I run the code I can see that it says: Loaded network /home/mona/computer_vision/py-faster-rcnn/data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel Browsing to faster_rcnn_models I have:

mona@pascal:~/computer_vision/py-faster-rcnn$ ls  /home/mona/computer_vision/py-faster-rcnn/data/faster_rcnn_models
VGG16_faster_rcnn_final.caffemodel  ZF_faster_rcnn_final.caffemodel

I downloaded this caffemodel for mscoco (is this the correct one?): mona@pascal:~/computer_vision/py-faster-rcnn/data/faster_rcnn_models$ wget www.cs.berkeley.edu/~rbg/faster-rcnn-data/coco_vgg16_faster_rcnn_final.caffemodel

How can I tell the code to use the .prototxt files from this directory?

mona@pascal:~/computer_vision/py-faster-rcnn/models/coco/VGG16/faster_rcnn_end2end$ ls
solver.prototxt  test.prototxt  train.prototxt

I have changed this line in mona@pascal:~/computer_vision/py-faster-rcnn$ vi lib/fast_rcnn/config.py

__C.MODELS_DIR = osp.abspath(osp.join(__C.ROOT_DIR, 'models', 'pascal_voc')) to: __C.MODELS_DIR = osp.abspath(osp.join(__C.ROOT_DIR, 'models', 'coco'))

and have the following in my:

mona@pascal:~/computer_vision/py-faster-rcnn/data/faster_rcnn_models$ ls
coco_vgg16_faster_rcnn_final.caffemodel  VGG16_faster_rcnn_final.caffemodel  ZF_faster_rcnn_final.caffemodel

I also changed the code in demo_no_bb.py to include this https://gist.github.com/monajalal/ae805bdfc54fa0d855efa4722c6d6cb0

 58 NETS = {'coco' : ('COCO',
 59                   'coco_vgg16_faster_rcnn_final.caffemodel'),
 60 
 61         'vgg16': ('VGG16',
 62                   'VGG16_faster_rcnn_final.caffemodel'),
 63         'zf': ('ZF',
 64                   'ZF_faster_rcnn_final.caffemodel')}

are lines 58 and 59 correct? When I run this new code, I get this error:

mona@pascal:~/computer_vision/py-faster-rcnn$ ./tools/demo_no_bb.py 
Traceback (most recent call last):
  File "./tools/demo_no_bb.py", line 145, in <module>
    net = caffe.Net(prototxt, caffemodel, caffe.TEST)
RuntimeError: Could not open file /home/mona/computer_vision/py-faster-rcnn/models/coco/VGG16/faster_rcnn_alt_opt/faster_rcnn_test.pt

I am not sure how to fix this?

mona@pascal:~/computer_vision/py-faster-rcnn/models/coco/VGG16$ ls
faster_rcnn_end2end  fast_rcnn

mona@pascal:~/computer_vision/py-faster-rcnn/models/coco/VGG16/faster_rcnn_end2end$ ls
solver.prototxt  test.prototxt  train.prototxt

so I have 76 classes (from 81 mscoco categories) and I have changed all the places that were 81 to 76:

mona@pascal:~/computer_vision/py-faster-rcnn/models/coco/VGG16/faster_rcnn_end2end$ grep -irn 76 *
test.prototxt:549:    num_output: 76
train.prototxt:11:    param_str: "'num_classes': 76"
train.prototxt:517:    param_str: "'num_classes': 76"
train.prototxt:589:    num_output: 76

Additionally these are two of the other things I tried: https://gist.github.com/monajalal/b7af9236f4e7d4af047b8fa58901d79c

[twtygqyy]

Check the code of prototxt = os.path.join(cfg.MODELS_DIR, NETS[args.demo_net][0], 'faster_rcnn_alt_opt', 'faster_rcnn_test.pt') caffemodel = os.path.join(cfg.DATA_DIR, 'faster_rcnn_models', NETS[args.demo_net][1]) to read the end2end prototxt

and the output parameters are not correct

layer {
  name: "cls_score"
  type: "InnerProduct"
  bottom: "fc7"
  top: "cls_score"
  inner_product_param {
    num_output: $NUMBER_OF_CLASSES + 1 (BACKGROUND)
  }
}
layer {
  name: "bbox_pred"
  type: "InnerProduct"
  bottom: "fc7"
  top: "bbox_pred"
  inner_product_param {
    num_output: 4* ($NUMBER_OF_CLASSES + 1 (BACKGROUND))
  }
}

[sidgan]

The bash files that you are running are calling the solver.prototxt. The paths to the datasets and the train.prototxt are to be changed here.

[zhan-xu]

I think I have successfully run COCO-VGG model with the following steps:

1. download the model from www.cs.berkeley.edu/~rbg/faster-rcnn-data/coco_vgg16_faster_rcnn_final.caffemodel, put it in /FASTER_RCNN_ROOT_FOLDER/data/faster_rcnn_models/

2.Copy test.prototxt from /FASTER_RCNN_ROOT_FOLDER/models/coco/VGG16/faster_rcnn_end2end/ to /FASTER_RCNN_ROOT_FOLDER/models/pascle_voc/VGG16/. Rename test.prototxt as coco_test.pt

3.in demo.py, modify: prototxt = os.path.join(cfg.MODELS_DIR,NETS[args.demo_net][0],'faster_rcnn_alt_opt','coco_test.pt') caffemodel = '/FASTER_RCNN_ROOT_FOLDER/data/faster_rcnn_models/coco_vgg16_faster_rcnn_final.caffemodel'

4.Near the head of demo.py, modify: CLASSES = ('background', 'person', 'bicycle', 'car','motorcycle','airplane','bus','train', 'truck', 'boat','traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench','bird','cat','dog','horse','sheep', 'cow', 'elephant','bear','zebra','giraffe','hat','umbrella', 'handbag','tie','suitcase', 'frisbee','skis','snowboard','sports ball','kite', 'baseball bat','baseball glove','skateboard','surfboard','tennis racket', 'bottle','wine glass','cup','fork','knife','spoon','bowl','banana','apple','sandwich', 'orange','broccoli','carrot','hot dog','pizza','donut','cake','chair','couch','potted plant', 'bed','dining table','window','tv','laptop','mouse','remote','keyboard','cell phone','microwave', 'oven', 'sink','refrigerator','blender','book','clock','vase','scissors','teddy bear','hair drier','tooth brush')

I cannot ganrantee everything, but at least these work for me.

[vanjiminster]

I created a docker image with a working demo using the Coco model. https://hub.docker.com/r/jimmyli/faster-rcnn-gpu/

[thuwangzhe]

@enderhsu The link of coco-vgg model you given above is invalid now, could you tell how to download this model now? Or could you send it to me? my email address is sjtuwangzhe2015@163.com, thank you very much!