Results for PredCls and PhrCls

rowanz commented 6 years ago

Hi Yikang,

I'm looking to replicate the results for the other visual genome scene graph evaluation modes. To get the test results under your evaluation, I would need to run something like the following, right?

        if args.mode == 'sggen':
            total_cnt_t, rel_cnt_correct_t = net.evaluate(
                im_data, im_info, gt_objects.numpy()[0], gt_relationships.numpy()[0], gt_regions.numpy()[0],
                top_Ns = top_Ns, nms=True)
        elif args.mode == 'phrcls':
            total_cnt_t, rel_cnt_correct_t = net.evaluate(
                im_data, im_info, gt_objects.numpy()[0], gt_relationships.numpy()[0], gt_regions.numpy()[0],
                top_Ns = top_Ns, nms=False, use_gt_boxes=True, use_gt_regions=False)
        elif args.mode == 'predcls':
            total_cnt_t, rel_cnt_correct_t = net.evaluate(
                im_data, im_info, gt_objects.numpy()[0], gt_relationships.numpy()[0], gt_regions.numpy()[0],
                top_Ns = top_Ns, nms=False, use_gt_boxes=True, use_gt_regions=False, only_predicate=True)

I had to change a couple of things too:

Hierarchical_Descriptive_Model.evaluate throws an error when use_gt_boxes=True because im_info is a 1 x 3 tensor. I got the best results when I uncommented the division by the image scale (which makes sense as then the GT boxes are at the same scale as the ROI proposals), can you confirm that e.g. gt_boxes_object = gt_objects[:, :4] is right?
https://github.com/yikang-li/MSDN/blob/master/faster_rcnn/MSDN.py#L118 seems like it contains a bug, because only the top couple of ROIs are overwritten. Can you confirm that it should be changed to object_rois = object_rois_gt?

However, even when I did these things, I can't match your paper results for PredCls and PhrCls. For PredCls for instance, I get around 37% R@50 and 46% R@50. Is there something else you did to get these numbers?

Thanks! -Rowan

yikang-li commented 6 years ago

Yes, there is something wrong with this part. When releasing the code, we guarantee SGGen part.

We will check our previous checkpoints for more details. If any updates, we will get you informed.

junhocho commented 6 years ago

Looking forward to try predcls mode.

akira-l commented 5 years ago

any updates in code ？ I am still not able to find any evalution code in predcls?

yikang-li / MSDN

Results for PredCls and PhrCls #4