Open sak-18 opened 4 years ago
I have the same problem before. Are you loading the correct model? It should looks like
2020-07-16 14:19:09,686 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from /upload_causal_motif_sgdet/model_0028000.pth
When I load the wrong model, the result looks like this
After I changed to
/upload_causal_motif_sgdet/model_0028000.pth
, the result looks like this
@Opdoop thanks a lot. Seems like I was giving the wrong model. Will get back to you soon and let you know if that was the case.
@Opdoop you were right I noticed 2020-07-16 18:11:46,145 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from catalog://ImageNetPretrained/FAIR/20171220/X-101-32x8d
path to the pretrained detector while running inspite of mentioning the correct path. Was this the same error? If so how did you solve it.
MODEL.PRETRAINED_DETECTOR_CKPT set to /upload_causal_motif_sgdet and modify last_checkpoint
to your path. https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/issues/7#issue-578052449
@Opdoop getting slightly better results after the change mentioned. Would like to visualize and see once. Was your image visualization done on a custom image? If so how could you manage to do it only with custom_prediction.pytorch
and without any annotations using visualize_PredCls_and_SGCls.ipynb
without using visual_info.json
? Looked here #53.
Yes, it's custom image. I thinkvisual_info.json
is for mapping bbox index in custom_prediction.pytorch
with image file name. So I just changed this __getitem__
function to not return index but image file name.
My change is like this:
def __getitem__(self, index):
#if self.split == 'train':
# while(random.random() > self.img_info[index]['anti_prop']):
# index = int(random.random() * len(self.filenames))
if self.custom_eval:
img = Image.open(self.custom_files[index]).convert("RGB")
target = torch.LongTensor([-1])
if self.transforms is not None:
# img = F.to_tensor(img)
img, target = self.transforms(img, target)
## CHANGE
file_name = self.custom_files[index].split('/')[-1]
return img, target, file_name
#return img, target, index
And visualization code is modified from visualization/2.visualize_SGDet.ipynb Below is the script I used, which may be useful for you.
from PIL import Image, ImageDraw
import json
import torch
import pdb
data = torch.load('./upload_causal_motif_sgdet/custom_prediction.pytorch')
img_root_path = './temp_images/'
vocab_file = json.load(open('./datasets/vg/VG-SGG-dicts-with-attri.json'))
idx2label = vocab_file['idx_to_label'] # label is object,size 150
idx2pred = vocab_file['idx_to_predicate'] # predicate is relation,size 50
idx2attr = vocab_file['idx_to_attribute'] # attribute, size 200
def get_info(prediction):
'''
Input SGDet bbox, get string result.
:param prediction: bbox.files() -> ['pred_labels', 'pred_scores', 'pred_attributes', 'rel_pair_idxs', 'pred_rel_scores', 'pred_rel_labels']
:return: pred_rels: (obj1, rela, obj2) list of tuple
pred_obj_attr: (obj, attr) list of tuple
'''
## get obj
pred_obj = [idx2label[str(i)] for i in prediction.get_field('pred_labels').tolist()] # obj index to label
## get relation
pred_rel_label = prediction.get_field('pred_rel_scores') # relation score, shape of (nbr_predition, 51)
pred_rel_label[:,0] = 0 # (nbr_predition, 0)
pred_rel_score, pred_rel_label = pred_rel_label.max(-1) # pred_rel_score -> score,pred_rel_label -> max index position
## concate obj - rela - obj
pred_rel_pair = prediction.get_field('rel_pair_idxs').tolist() # obj1 * obj2 matrix
pred_rels = [(pred_obj[i[0]], idx2pred[str(j)], pred_obj[i[1]]) for i, j in
zip(pred_rel_pair, pred_rel_label.tolist())] # concate to obj1-rela-obj2 triplets
## get attribute
# pred_attr = prediction.get_field('pred_attributes').to(torch.float32) # gey attributes score, shape of (nbr_predition, 201)
# pred_attr[:,0] = 0 # (nbr_predition, 0)
# pred_attr_score, pred_attr_index = pred_attr.max(-1) #
# pred_attr_label = [idx2attr[str(i)] for i in pred_attr_index.tolist()] #
#
# ## concate obj - attr
# pred_obj_attr = [(obj, attr) for obj, attr in zip(pred_obj, pred_attr_label)] # obj-attr
return pred_rels, pred_obj #pred_obj_attr # pred_rel_score, pred_rel_label
def draw_single_box(pic, box, color='red', draw_info=None):
draw = ImageDraw.Draw(pic)
x1,y1,x2,y2 = int(box[0]), int(box[1]), int(box[2]), int(box[3])
draw.rectangle(((x1, y1), (x2, y2)), outline=color)
if draw_info:
draw.rectangle(((x1, y1), (x1+50, y1+10)), fill=color)
info = draw_info
draw.text((x1, y1), info)
def draw_image(img_filename, prediction, pred_obj_attr, save_img=True):
img_path = '{}/{}'.format(img_root_path, img_filename)
pic = Image.open(img_path)
image_width, image_height = pic.size
prediction = prediction.resize((image_width, image_height)) # image resize
boxes = prediction.bbox
num_obj = boxes.shape[0]
for i in range(num_obj):
info = str(pred_obj_attr[i])
# info = '{}-{}'.format(pred_obj_attr[i][0], pred_obj_attr[i][1])
draw_single_box(pic, boxes[i], draw_info=info)
if i > 20:
break
if save_img:
pic.save('./tmp/Boxes_{}'.format(img_filename))
return None
for img_filename, bbox in data.items():
pred_rels, pred_obj_attr = get_info(bbox)
draw_image(img_filename, bbox, pred_obj_attr)
@Opdoop hello,my work on this also meet the same problem,but I use 3.visualize_custom_SGDet.ipynb ,so I want to cantact you on qq and I dont know where my fault is. Could you give me your qq?
❓ Questions and Help
Hi, @KaihuaTang Thanks for the wonderful repo, a very friendly repository. I have a small query. I have used SGdet pretrained model to get predictions on custom images which generated a
custom-prediction.pytorch
. I have usedBoxList.get('relation_pair_idx')
to get object pairs andBoxList.get('pred_rel_labels')
to get the predicates. But since these are numbers I used VG-SGG-dicts-with-attri.json to get actual names of the objects and relations by using "idx_to_label" and "idx_to_predicate" respectively. But the results seem to be random. In some cases I get a 0 as a label with BoxList.get('relation_pair_idx') which does not have a corresponding entry in the json file. Am I missing something? How do we get the names of objects and predicates? Is there a later version ofVG-SGG-dicts-with-attri.json
.