Net accuracy

[ericromanenghi]

How can i get the net accuracy?

There is layer like "test_on_test" and "test_on_train" like in the regular caffe?

Could you help me please?

[JoeLoveFannie]

Try test_net.py

[deboc]

Well I'm not sure but test_net.py seems to need pre-computed selective search to run properly.

And I think the issue is more about testing the accuracy during the training phase (Otherwise it's still the issue I encounter).

In other words, what is the equivalent of caffe's solver.prototxt "test_iter" and "test_interval" in py-faster-rcnn ?

[ericromanenghi]

Yes, i need to test during training, just like the regular caffe.

[ericromanenghi]

Could you solve it @deboc?

[deboc]

I haven't tried yet. Until now I hoped it was some misconfiguration on my side.

So anyone here train faster-rcnn without any accuracy-over-iterations feedback ?

[ericromanenghi]

I was reading the issues in this repo and in Fast RCNN repo, and it seems like there is not test on train in the python layer, and the only test is after training, using test_net.py

But if someone found something different, i'll glad to know about that.

[leejiajun]

@deboc @JoeLoveFannie I add an AccuracyLayer, but failed.

Check failed: param_.test_iter_size() == num_test_nets (1 vs. 0) test_iter must be specified for each test network.

layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "cls_score"
  bottom: "labels"
  top: "accuracy"
  include {
    phase: TEST
  }
  accuracy_param {
    top_k: 5
  }
}

[ericromanenghi]

I did the test using the script test_net.py.

But i still can't make a test during train. I was reading the issues (opened and closed) in this repo and in fast RCNN, and i couldn't find something useful.

[leejiajun]

@eternautaCAT me too.

[agatabcool]

Has anyone solved this? when I tried test_net.py I get a memory error after about 100 images.

[ericromanenghi]

I'm doing the test with test_net.py, but I have to test after training, because I have not enough memory to train and test at the same time.

[agatabcool]

@ericromanenghi right- this is what I did. The exact command: ./tools/test_net.py --def models/CaffeNet/test.prototxt --net data/fast_rcnn_models/caffenet_fast_rcnn_iter_40000.caffemodel --imdb imagenet_train

where imagenet_train is the name of the python file that generates my imdb - the exact way in which it was done in training. (As an fyi, my training set is 900 images)

[ericromanenghi]

Try adding

--cfg experiments/cfgs/faster_rcnn_end2end.yml

[agatabcool]

This didn't work. I finally got it to work by switching to a server with more GPU memory, however now when i run test_net - it throws an error at the last line:

Evaluating detections
Traceback (most recent call last):
  File "./tools/test_net.py", line 81, in <module>
    test_net(net, imdb)
  File "tools/../lib/fast_rcnn/test.py", line 327, in test_net
    imdb.evaluate_detections(nms_dets, output_dir)
  File "tools/../lib/datasets/imdb.py", line 92, in evaluate_detections
    raise NotImplementedError
NotImplementedError

This makes complete sense because indeed the original code doesn't have an implementation --- are we supposed to just implement one ourselves?

[ericromanenghi]

Yes and no. imdb is just an interface. Actually, the implementation of the functions is in lib/datasets/pascal_voc.py for example for pascal VOC. When you want to use py-faster in your own dataset, you have to make a python file similar to pascal_voc, put it in that folder, and add it to factory.py. If you take a look of the code of pascal_voc.py, you'll see the function evaluate_detections implemented.

I hope help you, but if you read the code in factory.py, pascal_voc.py and imdb.py trying to follow the logic, you will understand better.

[jeanlancel]

Regarding the original topic of testing during train time, it is possible to rewrite the training loop in lib/fast_rcnn/train.py under the function train_model(self, max_iters) so that the while loop can call a testing loop every x iterations, without considering any hard ware constraints (basically just like how we test and train at the same time for any other CNN on caffe). In terms of the memory issue, I'm under the impression that caffe is only VRAM intensive? If so then declaring the testing net in CPU mode might work, albeit a t a much slower rate. The call to the testing loop can be done parallelly and the training thread won't have to wait for the testing to complete.

[DeepestNet]

Yes, I agree with @jeanlancel.