Predictions on SGdet with VG-SGG-dicts-with-attri.json

[sak-18]

❓ Questions and Help

Hi, @KaihuaTang Thanks for the wonderful repo, a very friendly repository. I have a small query. I have used SGdet pretrained model to get predictions on custom images which generated a custom-prediction.pytorch. I have used BoxList.get('relation_pair_idx') to get object pairs and BoxList.get('pred_rel_labels') to get the predicates. But since these are numbers I used VG-SGG-dicts-with-attri.json to get actual names of the objects and relations by using "idx_to_label" and "idx_to_predicate" respectively. But the results seem to be random. In some cases I get a 0 as a label with BoxList.get('relation_pair_idx') which does not have a corresponding entry in the json file. Am I missing something? How do we get the names of objects and predicates? Is there a later version of VG-SGG-dicts-with-attri.json.

[Opdoop]

I have the same problem before. Are you loading the correct model? It should looks like 2020-07-16 14:19:09,686 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from /upload_causal_motif_sgdet/model_0028000.pth

[Opdoop]

When I load the wrong model, the result looks like this After I changed to /upload_causal_motif_sgdet/model_0028000.pth, the result looks like this

[sak-18]

@Opdoop thanks a lot. Seems like I was giving the wrong model. Will get back to you soon and let you know if that was the case.

[sak-18]

@Opdoop you were right I noticed 2020-07-16 18:11:46,145 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from catalog://ImageNetPretrained/FAIR/20171220/X-101-32x8d path to the pretrained detector while running inspite of mentioning the correct path. Was this the same error? If so how did you solve it.

[Opdoop]

MODEL.PRETRAINED_DETECTOR_CKPT set to /upload_causal_motif_sgdet and modify last_checkpoint to your path. https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/issues/7#issue-578052449

[sak-18]

@Opdoop getting slightly better results after the change mentioned. Would like to visualize and see once. Was your image visualization done on a custom image? If so how could you manage to do it only with custom_prediction.pytorch and without any annotations using visualize_PredCls_and_SGCls.ipynb without using visual_info.json? Looked here #53.

[Opdoop]

Yes, it's custom image. I thinkvisual_info.json is for mapping bbox index in custom_prediction.pytorch with image file name. So I just changed this __getitem__ function to not return index but image file name.

https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/54655eb4d6485825b6d902cf0d1b3c34436a635f/maskrcnn_benchmark/data/datasets/visual_genome.py#L71

My change is like this:

    def __getitem__(self, index):
        #if self.split == 'train':
        #    while(random.random() > self.img_info[index]['anti_prop']):
        #        index = int(random.random() * len(self.filenames))
        if self.custom_eval:
            img = Image.open(self.custom_files[index]).convert("RGB")
            target = torch.LongTensor([-1])
            if self.transforms is not None:
                # img = F.to_tensor(img)
                img, target = self.transforms(img, target)
            ## CHANGE 
            file_name = self.custom_files[index].split('/')[-1]
            return img, target, file_name
            #return img, target, index

And visualization code is modified from visualization/2.visualize_SGDet.ipynb Below is the script I used, which may be useful for you.

from PIL import Image, ImageDraw
import json
import torch
import pdb

data = torch.load('./upload_causal_motif_sgdet/custom_prediction.pytorch')

img_root_path = './temp_images/'

vocab_file = json.load(open('./datasets/vg/VG-SGG-dicts-with-attri.json'))
idx2label = vocab_file['idx_to_label'] # label is object，size 150
idx2pred = vocab_file['idx_to_predicate'] # predicate is relation，size 50 
idx2attr = vocab_file['idx_to_attribute'] # attribute， size 200 

def get_info(prediction):
    '''
    Input SGDet bbox, get string result.
    :param prediction: bbox.files() -> ['pred_labels', 'pred_scores', 'pred_attributes', 'rel_pair_idxs', 'pred_rel_scores', 'pred_rel_labels']
    :return: pred_rels: (obj1, rela, obj2) list of tuple
            pred_obj_attr: (obj, attr) list of tuple
    '''
    ## get obj
    pred_obj = [idx2label[str(i)] for i in prediction.get_field('pred_labels').tolist()] # obj index to label

    ## get relation
    pred_rel_label = prediction.get_field('pred_rel_scores') #  relation score, shape of (nbr_predition, 51)
    pred_rel_label[:,0] = 0 # （nbr_predition, 0) 
    pred_rel_score, pred_rel_label = pred_rel_label.max(-1) # pred_rel_score -> score，pred_rel_label -> max index position

    ## concate obj - rela - obj
    pred_rel_pair = prediction.get_field('rel_pair_idxs').tolist() # obj1 * obj2 matrix
    pred_rels = [(pred_obj[i[0]], idx2pred[str(j)], pred_obj[i[1]]) for i, j in
                 zip(pred_rel_pair, pred_rel_label.tolist())] # concate to  obj1-rela-obj2 triplets

    ## get attribute
    # pred_attr = prediction.get_field('pred_attributes').to(torch.float32) # gey attributes score, shape of (nbr_predition, 201)
    # pred_attr[:,0] = 0 # （nbr_predition, 0) 
    # pred_attr_score, pred_attr_index = pred_attr.max(-1) # 
    # pred_attr_label = [idx2attr[str(i)] for i in pred_attr_index.tolist()] # 
    #
    # ## concate obj - attr
    # pred_obj_attr = [(obj, attr) for obj, attr in zip(pred_obj, pred_attr_label)] # obj-attr

    return pred_rels, pred_obj #pred_obj_attr # pred_rel_score, pred_rel_label

def draw_single_box(pic, box, color='red', draw_info=None):
    draw = ImageDraw.Draw(pic)
    x1,y1,x2,y2 = int(box[0]), int(box[1]), int(box[2]), int(box[3])
    draw.rectangle(((x1, y1), (x2, y2)), outline=color)
    if draw_info:
        draw.rectangle(((x1, y1), (x1+50, y1+10)), fill=color)
        info = draw_info
        draw.text((x1, y1), info)

def draw_image(img_filename, prediction, pred_obj_attr, save_img=True):
    img_path = '{}/{}'.format(img_root_path, img_filename)
    pic = Image.open(img_path)
    image_width, image_height = pic.size
    prediction = prediction.resize((image_width, image_height)) # image resize
    boxes = prediction.bbox
    num_obj = boxes.shape[0]
    for i in range(num_obj):
        info = str(pred_obj_attr[i])
        # info = '{}-{}'.format(pred_obj_attr[i][0], pred_obj_attr[i][1])
        draw_single_box(pic, boxes[i], draw_info=info)
        if i > 20:
            break
    if save_img:
        pic.save('./tmp/Boxes_{}'.format(img_filename))

    return None

for img_filename, bbox in data.items():
    pred_rels, pred_obj_attr = get_info(bbox)
    draw_image(img_filename, bbox, pred_obj_attr)

[pcyyo]

@Opdoop hello，my work on this also meet the same problem，but I use 3.visualize_custom_SGDet.ipynb ，so I want to cantact you on qq and I dont know where my fault is. Could you give me your qq？