search

yangxuntu / SGAE

220 stars 48 forks source link

How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? #7

Open happynewya opened 5 years ago

happynewya commented 5 years ago

for example the coco_pred_sg.zip: hope there are more explicit descriptions on it, thanks

Out[32]:
array([[  0.,   9., 408.],
       [  0.,  14., 425.],
       [  0.,  26., 425.],
       [  1.,  14., 408.],
       [  1.,  26., 425.],
       [  2.,  22., 410.],
       [  3.,  13., 429.],
       [  3.,  14., 408.],
       [  5.,   7., 410.],
       [  5.,  26., 408.],
       [  7.,  14., 408.],
       [  7.,  26., 408.],
       [ 13.,  14., 463.],
       [ 14.,  26., 408.],
       [ 18.,  20., 408.]])
ravissj4 commented 5 years ago

Also, in the readme you said that 'coco_pred_sg.zip' and 'coco_spice_sg2.zip' are scene graphs of the image and sentences respectively. But, there is a 3rd folder - coco_img_sg, could you please tell us what does it contain ?

yangxuntu commented 5 years ago

For relationship matrix [ 0., 9., 408.], it means the relationship between the 0-th box and the 9-th box is relation 'dict[408]', if 'dict[408]' is 'on', then the corresponding relationship is 'on'.

For attribute matrix [0,1,2], it means that this bounding box own three attributes which are 'dict[0]', 'dict[1]' and 'dict[2]'.

Coco_img_sg is the scene graph generated by using image, not from sentences.

HYPJUDY commented 5 years ago

I also spent some time figuring what these files mean until I found show_sg.py provided by the author. I modified some lines for better visualization and post it here for your reference.

from __future__ import print_function
import argparse
import numpy as np

parser = argparse.ArgumentParser()
# Input paths
parser.add_argument('--id', type=str, default='0',
                help='id of image')
parser.add_argument('--mode', type=str,  default='sen',
                help='image or sen')

opt = parser.parse_args()
if opt.mode == 'sen':
    sg_dict = np.load('spice_sg_dict2.npz')['spice_dict'][()]
    sg_dict = sg_dict['ix_to_word']
    folder = 'coco_spice_sg2/'
else:
    sg_dict = np.load('coco_pred_sg_rela.npy')[()]
    sg_dict = sg_dict['i2w']
    folder = 'coco_img_sg/'

sg_path = folder + opt.id + '.npy'
sg_use = np.load(sg_path)[()]
if opt.mode == 'sen':
    rela = sg_use['rela_info']
    obj_attr = sg_use['obj_info']
else:
    rela = sg_use['rela_matrix']
    obj_attr = sg_use['obj_attr']
N_rela = len(rela)
N_obj = len(obj_attr)
for i in range(N_obj):
    if opt.mode == 'sen':
        print('obj #{0}'.format(i), end = ': ')
        if len(obj_attr[i]) >= 2:
            print ('(', end = '')
            for j in range(len(obj_attr[i])-1):
                print('{0} '.format(sg_dict[obj_attr[i][j + 1]]), end = '')
            print (') ', end = '')
        print(sg_dict[obj_attr[i][0]])
    else:
        print('obj #{0}'.format(i), end = ': ')  # maybe it means 'bounding box' but not 'object'
        N_attr = 3
        for j in range(N_attr - 1):
            print('{0} {1}, '.format(sg_dict[obj_attr[i][j + 4]],\
                sg_dict[obj_attr[i][j+1]]), end = '')
        j = N_attr - 1
        print('{0} {1}'.format(sg_dict[obj_attr[i][j + 4]],\
            sg_dict[obj_attr[i][j+1]]))

for i in range(N_rela):
    obj_idx = 0 if opt.mode == 'sen' else 1
    sbj = sg_dict[ int(obj_attr[int(rela[i][0])][obj_idx]) ]
    obj = sg_dict[ int(obj_attr[int(rela[i][1])][obj_idx]) ]
    rela_name = sg_dict[rela[i][2]]
    print('rel #{3}: {0}-{1}-{2}'.format(sbj,rela_name,obj,i))

Let's take image COCO_val2014_000000391895.jpg as an example. COCO_val2014_000000391895

GT of image_id #391895
A man with a red helmet on a small moped on a dirt road. 
Man riding a motor bike on a dirt road on the countryside.
A man riding on the back of a motorcycle.
A dirt path with a young person on a motor bike rests to the foreground of a verdant area with a bridge and a background of cloud-wreathed mountains. 
A man in a red shirt and a red hat is on a motorcycle on a hill side.

Put those npy and npz files under corresponding paths specified in my modified show_sg.py and run it with commands python show_sg.py --mode img --id 391895 and python show_sg.py --mode sen --id 391895. The outputs are as follows:

obj #0: red man, stand person, wear bike
obj #1: red helmet, red  hat, black cap
obj #2: blue bike, white motorcycle, small bicycle
obj #3: white sky, cloudy cloud, blue mountain
obj #4: green grass, brown grind, small road
obj #5: brown road, green grind, large dirt
obj #6: large mountain, tall background, green tree
obj #7: black wheel, dirty tire, brown part
obj #8: stand man, sit person, wear guy
obj #9: red shirt, orange jacket, red  sweater
obj #10: green mountain, large hill, tall tree
obj #11: brown road, wet grind, white dirt
obj #12: white mountain, large hill, tall tree
obj #13: brown shoe, tan foot, leather boot
obj #14: green tree, large mountain, tall hill
obj #15: green rock, small plant, large stone
obj #16: blue bike, white motorcycle, small bicycle
obj #17: white wheel, tan tire, brown pipe
obj #18: wooden pillar, concrete post, white pole
obj #19: brown bridge, green track, metal tree
obj #20: green grass, tall bush, short plant
obj #21: wooden fence, wood wall, stone bridge
obj #22: blue short, white jean, black pant
obj #23: white mirror, silver light, small sign
obj #24: green bush, leafy tree, large plant
obj #25: green grass, tall bush, short plant
obj #26: tall mountain, green hill, large tree
obj #27: brown road, wet dirt, green grind
obj #28: metal bridge, wooden track, brown rail
obj #29: sit bench, white people, stand person
obj #30: green tree, leafy bush, tall leaf
obj #31: green tree, large bush, leafy leaf
obj #32: red man, white person, stand men
obj #33: wooden post, wood pole, concrete pillar
obj #34: brown bag, red basket, black luggage
obj #35: blue bike, white motorcycle, yellow wheel
obj #36: metal bridge, brown track, long rail
rel #0: man-on-grass
rel #1: man-on-road
rel #2: man-in-mountain
rel #3: man-wear-mountain
rel #4: man-on-fence
rel #5: man-on-mountain
rel #6: bike-on-grass
rel #7: bike-under-man
rel #8: bike-ha-bike
rel #9: bike-ha-bike
rel #10: sky-in-mountain
rel #11: sky-above-mountain
rel #12: sky-above-mountain
rel #13: grass-next to-bike
rel #14: grass-ha-grass
rel #15: grass-on-road
rel #16: grass-near-bike
rel #17: road-on-road
rel #18: mountain-ha-mountain
rel #19: shirt-on-man
rel #20: mountain-in-tree
rel #21: mountain-on-mountain
rel #22: mountain-on-mountain
rel #23: mountain-wear-man
rel #24: tree-behind-bridge
rel #25: bike-in front of-fence
rel #26: bike-on-bike
rel #27: grass-on side of-road
rel #28: mountain-behind-man
rel #29: tree-ha-tree

and 

obj #0: (red ) hat
obj #1: small
obj #2: bridge
obj #3: (red ) shirt
obj #4: countryside
obj #5: back
obj #6: (dirt ) path
obj #7: (red ) helmet
obj #8: mountain
obj #9: (young ) person
obj #10: (hill ) side
obj #11: (verdant ) area
obj #12: man
obj #13: background
obj #14: (dirt ) road
obj #15: foreground
obj #16: (motor ) motorcycle/bike
rel #0: path-with-person
rel #1: path-rest to-foreground
rel #2: back-of-motorcycle/bike
rel #3: helmet-on-small
rel #4: person-on-motorcycle/bike
rel #5: foreground-of-area
rel #6: foreground-of-background
rel #7: motorcycle/bike-on-side
rel #8: man-ride-motorcycle/bike
rel #9: man-on-motorcycle/bike
rel #10: man-in-shirt
rel #11: man-ride on-back
rel #12: man-in-hat
rel #13: man-with-helmet
rel #14: man-ride on-road
rel #15: area-with-bridge
rel #16: road-on-countryside
rel #17: background-of-mountain

Note that the index 0 in the following lines of original code is obviously wrong for image mode because the 0 index indicating the number of objects:

for i in range(N_rela):
    sbj = sg_dict[ int(obj_attr[int(rela[i][0])][0]) ]
    obj = sg_dict[ int(obj_attr[int(rela[i][1])][0]) ]

and I modified it to

for i in range(N_rela):
    obj_idx = 0 if opt.mode == 'sen' else 1
    sbj = sg_dict[ int(obj_attr[int(rela[i][0])][obj_idx]) ]
    obj = sg_dict[ int(obj_attr[int(rela[i][1])][obj_idx]) ]

But I'm not sure whether index 1 is correct because there are three attributes for each object (maybe it means bounding box but not object?). I used the first one. Is it right?

yangxuntu commented 5 years ago

Thank you very much, I will check the confusion of index next week. There are so many versions of scene graphs of this research, and sometimes I need to carefully check the codes for understanding what I was trying to do before.

HYPJUDY commented 5 years ago

Thanks, you really did a hard and good work!

HYPJUDY commented 5 years ago

Hi, thanks for you guidance for generating scene graphs for sentences. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back                ","helmet              ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","foreground          ","foreground          ","motorcycle\/bike     ","road                ","area                ","path                ","path                ","person              ","background          "],"attribute":["node: back                ","node: helmet              ","attr: red                 ","node: shirt               ","attr: red                 ","node: small               ","node: man                 ","node: side                ","attr: hill                ","node: foreground          ","node: motorcycle\/bike     ","attr: motor               ","node: road                ","attr: dirt                ","node: area                ","attr: verdant             ","node: path                ","attr: dirt                ","node: mountain            ","attr: cloud-wreathed      ","node: bridge              ","node: person              ","attr: young               ","node: hat                 ","attr: red                 ","node: background          ","node: countryside         "],"rela":["of                  ","on                  ","on                  ","ride                ","ride on             ","ride on             ","mope on             ","with                ","in                  ","in                  ","of                  ","of                  ","on                  ","on                  ","with                ","rest to             ","with                ","on                  ","of                  "],"object":["motorcycle\/bike     ","small               ","motorcycle\/bike     ","motorcycle\/bike     ","back                ","road                ","road                ","helmet              ","shirt               ","hat                 ","area                ","background          ","side                ","countryside         ","bridge              ","foreground          ","person              ","motorcycle\/bike     ","mountain            "]}}
yangxuntu commented 5 years ago

以下内容是如何处理sentence scene graph的： 1).create_coco_sg.py 通过spice中的parser方法来生成coco dataset中每个图片的scene graph 运行的时候要先进代码将coco_use设置为coco_train,然后把self-critical.pytorch/coco-caption/pycocoevalcap/spice/中的sg.json保存为spice_sg_train.json，然后把这个文件放到/data中去，同理把coco_use设置为coco_val，然后把生成的sg.json保存为spice_sg_val.json。

2).process_spice_sg.py 将spice_sg_val.json和spice_sg_train.json处理为按照image_id.npy形式保存在coco_spice_sg中。具体格式为： ssg['obj_info']:list,每一个list都代表着一个object以及其对应的attribute， 第一个元素是object name在dict中的index，后面的元素都是 attribute对应dict中的index。 ssg['rela_info']:numpy数组，每一行3个元素，分别为[sbj_id,obj_id,relation_id] 其中的sbj_id和obj_id对应于ssg['obj_info']中的obj list的index， relation_id为其在dict中的index。 还有处理完的dict可以保存为spice_sg_dict.npz. 我先生成了一个spice的字典，其中把很多词语正则化了一下，前9487个词和原始的词典一致，后面的为增补的词典。

3).process_sg_extend.py 生成适合attention extend model的数据,直接从之前的生成的数据中来改。 先生成一个字典，其中前9487个词和原始词典一致，然后加入sentence scene graph的字典，以及image scene graph的字典。 然后把image scene graph的数据改了就好了。也就是只改变cocobu_rela中的数据，然后把改变完的数据保存到cocobu_sg_img中去。 而对于sentence scene graph，其还是保存在coco_spice_sg中。 新的字典保存在sg_dict_extend.npz中。

对于coco_spice_sg2，其实就是字典更大了，比如coco_spice_sg中对应的字典是caption生成的字典时把小于等于5个词的词给删除了，然而coco_spice_sg2中对应的字典是caption生成的字典时把小于等于4个词的词给删除了.不同的字典对最后影响不是那么大。 因为最近比较忙，就直接写中文了。 最后还是谢谢:)

发件人: Yupan Huang notifications@github.com 发送时间: 2019年8月22日 21:21 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

Hi, thanks for you guidance for generating scene graphs for sentenceshttps://github.com/yangxuntu/SGAE#generating-scene-graphs. I've generated the sg.json file for my dataset. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back ","helmet ","man ","man ","man ","man ","man ","man ","man ","man ","foreground ","foreground ","motorcycle\/bike ","road ","area ","path ","path ","person ","background "],"attribute":["node: back ","node: helmet ","attr: red ","node: shirt ","attr: red ","node: small ","node: man ","node: side ","attr: hill ","node: foreground ","node: motorcycle\/bike ","attr: motor ","node: road ","attr: dirt ","node: area ","attr: verdant ","node: path ","attr: dirt ","node: mountain ","attr: cloud-wreathed ","node: bridge ","node: person ","attr: young ","node: hat ","attr: red ","node: background ","node: countryside "],"rela":["of ","on ","on ","ride ","ride on ","ride on ","mope on ","with ","in ","in ","of ","of ","on ","on ","with ","rest to ","with ","on ","of "],"object":["motorcycle\/bike ","small ","motorcycle\/bike ","motorcycle\/bike ","back ","road ","road ","helmet ","shirt ","hat ","area ","background ","side ","countryside ","bridge ","foreground ","person ","motorcycle\/bike ","mountain "]}}

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOREXTHPVDFQNU64BC3QF2HFFA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD45CEFA#issuecomment-523903508, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJEJUOTBTGIN5T3YC4TEXUTQF2HFFANCNFSM4IEKOZSQ.

HYPJUDY commented 5 years ago

Thanks for your detailed reply! It seems that process_spice_sg.py and process_sg_extend.py haven't been uploaded. Could you share them? BTW, where's the modifed code of spice for generating sg.json? There's 0 match of sg.json in your coco-caption/pycocoevalcap/spice. Maybe you modified the spice-1.0.jar for this purpose? May I know how to do it? Thanks!

yangxuntu commented 5 years ago

I send them to you via email, but it seems you not receive them, then I upload them into the github. I upload the spice-1.0.jar for creating sentence scene graph. The following are the instructions I noted when I write those codes: 通过spice中的parser方法来生成coco dataset中每个图片的scene graph 运行的时候要先进代码将coco_use设置为coco_train,然后把self-critical.pytorch/coco-caption/pycocoevalcap/spice/中的sg.json保存为spice_sg_train.json，然后把这个文件放到/data中去，同理把coco_use设置为coco_val，然后把生成的sg.json保存为spice_sg_val.json。

发件人: Yupan Huang notifications@github.com 发送时间: 2019年8月23日 13:34 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

Thanks for your detailed reply! It seems that process_spice_sg.py and process_sg_extend.py haven't been uploaded. Could you share them? BTW, where's the modifed code of spice for generating sg.json? There's 0 match of sg.json in your coco-caption/pycocoevalcap/spice. Maybe you modified the spice-1.0.jar for this purpose? May I know how to do it? Thanks!

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOX7U6JP4EPSPCPMD2DQF5ZFHA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD47FK4I#issuecomment-524178801, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJEJUOS7PVYE7VN444ANJVTQF5ZFHANCNFSM4IEKOZSQ.

HYPJUDY commented 5 years ago

Thanks! A kind remind: There are two identical spice-1.0.jar under folder SGAE and SGAE\coco-caption\pycocoevalcap\spice respectively so maybe you can delete the first one to save space (18.8M) : )

yangxuntu commented 5 years ago

Ok.

发件人: Yupan Huang notifications@github.com 发送时间: 2019年8月23日 21:05 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

Thanks! A kind remind: There are two identical spice-1.0.jar under folder SGAE and SGAE\coco-caption\pycocoevalcap\spice respectively so maybe you can delete the first one to save space (18.8M) : )

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOWSM2Y4XO4QZAJP4YDQF7OBNA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5AE24Q#issuecomment-524307826, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJEJUOX5DQGMQQZNV57J6LDQF7OBNANCNFSM4IEKOZSQ.

happynewya commented 5 years ago

Have you provided the bounding boxes of each image? Thank you!

thsunkid commented 5 years ago

Hi, thanks for you guidance for generating scene graphs for sentences. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back                ","helmet              ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","foreground          ","foreground          ","motorcycle\/bike     ","road                ","area                ","path                ","path                ","person              ","background          "],"attribute":["node: back                ","node: helmet              ","attr: red                 ","node: shirt               ","attr: red                 ","node: small               ","node: man                 ","node: side                ","attr: hill                ","node: foreground          ","node: motorcycle\/bike     ","attr: motor               ","node: road                ","attr: dirt                ","node: area                ","attr: verdant             ","node: path                ","attr: dirt                ","node: mountain            ","attr: cloud-wreathed      ","node: bridge              ","node: person              ","attr: young               ","node: hat                 ","attr: red                 ","node: background          ","node: countryside         "],"rela":["of                  ","on                  ","on                  ","ride                ","ride on             ","ride on             ","mope on             ","with                ","in                  ","in                  ","of                  ","of                  ","on                  ","on                  ","with                ","rest to             ","with                ","on                  ","of                  "],"object":["motorcycle\/bike     ","small               ","motorcycle\/bike     ","motorcycle\/bike     ","back                ","road                ","road                ","helmet              ","shirt               ","hat                 ","area                ","background          ","side                ","countryside         ","bridge              ","foreground          ","person              ","motorcycle\/bike     ","mountain            "]}}

Hi, I wonder how can you have this sg.json file and demo? In both create_coco_sg.py and spice.py I can't find any clues about the process that generates sg.json.

It seems like Spice.compute_score in create_coco_sg.py file just works as scoring function.

Btw, I found another repo can help to generate sentence graph in python: https://github.com/vacancy/SceneGraphParser . But I don't know does it work similar to the thing in this project.

yangxuntu commented 5 years ago

Try this file.

发件人: Thu Nguyen notifications@github.com 发送时间: 2019年9月17日 12:47 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

Hi, thanks for you guidance for generating scene graphs for sentenceshttps://github.com/yangxuntu/SGAE#generating-scene-graphs. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back ","helmet ","man ","man ","man ","man ","man ","man ","man ","man ","foreground ","foreground ","motorcycle\/bike ","road ","area ","path ","path ","person ","background "],"attribute":["node: back ","node: helmet ","attr: red ","node: shirt ","attr: red ","node: small ","node: man ","node: side ","attr: hill ","node: foreground ","node: motorcycle\/bike ","attr: motor ","node: road ","attr: dirt ","node: area ","attr: verdant ","node: path ","attr: dirt ","node: mountain ","attr: cloud-wreathed ","node: bridge ","node: person ","attr: young ","node: hat ","attr: red ","node: background ","node: countryside "],"rela":["of ","on ","on ","ride ","ride on ","ride on ","mope on ","with ","in ","in ","of ","of ","on ","on ","with ","rest to ","with ","on ","of "],"object":["motorcycle\/bike ","small ","motorcycle\/bike ","motorcycle\/bike ","back ","road ","road ","helmet ","shirt ","hat ","area ","background ","side ","countryside ","bridge ","foreground ","person ","motorcycle\/bike ","mountain "]}}

Hi, I wonder how can you have this sg.json file and demo? In both create_coco_sg.py and spice.py I can't find any clues about the process that generates sg.json.

It seems like Spice.compute_score in create_coco_sg.py file just works as scoring function.

Btw, I found a this project can help to generate sentence graph in python: https://github.com/vacancy/SceneGraphParser . But I don't know does it work similar to the thing in this project.

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOWQ3CKTTS225Y5TIHLQKBOP5A5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD63IXCQ#issuecomment-532056970, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJEJUOX345TSG3JADQTMLNLQKBOP5ANCNFSM4IEKOZSQ.

thsunkid commented 5 years ago

Try this file. …

Hi yang,

Thanks for your quick reply. I'd watched your lastest commit, but It seems we even don't have a way to extract sg.json file from scratch. create_coco_sg.py doesn't work.

JingJLiu commented 4 years ago

Hi，when I use creat_coco_sg.py to generate sg.json I got a problem.Do you know what's the problem?Thank you so much. ''' loading annotations into memory... 0:00:03.230310 creating index... Traceback (most recent call last): File "create_coco_sg.py", line 19, in coco_train = COCO(train_path) File "coco-caption/pycocotools/coco.py", line 76, in init self.createIndex() File "coco-caption/pycocotools/coco.py", line 93, in createIndex if self.dataset['type'] == 'instances': KeyError: 'type' '''

JingJLiu commented 4 years ago

Hi, thanks for you guidance for generating scene graphs for sentences. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back                ","helmet              ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","man                 ","foreground          ","foreground          ","motorcycle\/bike     ","road                ","area                ","path                ","path                ","person              ","background          "],"attribute":["node: back                ","node: helmet              ","attr: red                 ","node: shirt               ","attr: red                 ","node: small               ","node: man                 ","node: side                ","attr: hill                ","node: foreground          ","node: motorcycle\/bike     ","attr: motor               ","node: road                ","attr: dirt                ","node: area                ","attr: verdant             ","node: path                ","attr: dirt                ","node: mountain            ","attr: cloud-wreathed      ","node: bridge              ","node: person              ","attr: young               ","node: hat                 ","attr: red                 ","node: background          ","node: countryside         "],"rela":["of                  ","on                  ","on                  ","ride                ","ride on             ","ride on             ","mope on             ","with                ","in                  ","in                  ","of                  ","of                  ","on                  ","on                  ","with                ","rest to             ","with                ","on                  ","of                  "],"object":["motorcycle\/bike     ","small               ","motorcycle\/bike     ","motorcycle\/bike     ","back                ","road                ","road                ","helmet              ","shirt               ","hat                 ","area                ","background          ","side                ","countryside         ","bridge              ","foreground          ","person              ","motorcycle\/bike     ","mountain            "]}}

Hi，I see you have already use creat_coco_sg.py to generate sg.json .But I have some problem, Can you give me your Email?Thank you so much.

yangxuntu commented 4 years ago

What is your problem?

发件人: JingJ_Liu notifications@github.com 发送时间: 2019年10月11日 9:07 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

Hi, thanks for you guidance for generating scene graphs for sentenceshttps://github.com/yangxuntu/SGAE#generating-scene-graphs. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI.

{"391895":{"subject":["back ","helmet ","man ","man ","man ","man ","man ","man ","man ","man ","foreground ","foreground ","motorcycle\/bike ","road ","area ","path ","path ","person ","background "],"attribute":["node: back ","node: helmet ","attr: red ","node: shirt ","attr: red ","node: small ","node: man ","node: side ","attr: hill ","node: foreground ","node: motorcycle\/bike ","attr: motor ","node: road ","attr: dirt ","node: area ","attr: verdant ","node: path ","attr: dirt ","node: mountain ","attr: cloud-wreathed ","node: bridge ","node: person ","attr: young ","node: hat ","attr: red ","node: background ","node: countryside "],"rela":["of ","on ","on ","ride ","ride on ","ride on ","mope on ","with ","in ","in ","of ","of ","on ","on ","with ","rest to ","with ","on ","of "],"object":["motorcycle\/bike ","small ","motorcycle\/bike ","motorcycle\/bike ","back ","road ","road ","helmet ","shirt ","hat ","area ","background ","side ","countryside ","bridge ","foreground ","person ","motorcycle\/bike ","mountain "]}}

Hi，I see you have already use creat_coco_sg.py to generate sg.json .But I have some problem, Can you give me your Email?Thank you so much.

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOQH6DU7CXO2AB3VRO3QN7GWXA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6NOAA#issuecomment-540858112, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJEJUOQH7S7ZJDO5V6JOACTQN7GWXANCNFSM4IEKOZSQ.

JingJLiu commented 4 years ago

What is your problem? … ____ 发件人: JingJ_Liu notifications@github.com 发送时间: 2019年10月11日 9:07 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7) Hi, thanks for you guidance for generating scene graphs for sentenceshttps://github.com/yangxuntu/SGAE#generating-scene-graphs. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI. {"391895":{"subject":["back ","helmet ","man ","man ","man ","man ","man ","man ","man ","man ","foreground ","foreground ","motorcycle\/bike ","road ","area ","path ","path ","person ","background "],"attribute":["node: back ","node: helmet ","attr: red ","node: shirt ","attr: red ","node: small ","node: man ","node: side ","attr: hill ","node: foreground ","node: motorcycle\/bike ","attr: motor ","node: road ","attr: dirt ","node: area ","attr: verdant ","node: path ","attr: dirt ","node: mountain ","attr: cloud-wreathed ","node: bridge ","node: person ","attr: young ","node: hat ","attr: red ","node: background ","node: countryside "],"rela":["of ","on ","on ","ride ","ride on ","ride on ","mope on ","with ","in ","in ","of ","of ","on ","on ","with ","rest to ","with ","on ","of "],"object":["motorcycle\/bike ","small ","motorcycle\/bike ","motorcycle\/bike ","back ","road ","road ","helmet ","shirt ","hat ","area ","background ","side ","countryside ","bridge ","foreground ","person ","motorcycle\/bike ","mountain "]}} Hi，I see you have already use creat_coco_sg.py to generate sg.json .But I have some problem, Can you give me your Email?Thank you so much. ― You are receiving this because you commented. Reply to this email directly, view it on GitHub<#7?email_source=notifications&email_token=AJEJUOQH6DU7CXO2AB3VRO3QN7GWXA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6NOAA#issuecomment-540858112>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJEJUOQH7S7ZJDO5V6JOACTQN7GWXANCNFSM4IEKOZSQ.

Thanks for your quick reply.The problem is: ''' loading annotations into memory... 0:00:03.230310 creating index... Traceback (most recent call last): File "create_coco_sg.py", line 19, in coco_train = COCO(train_path) File "coco-caption/pycocotools/coco.py", line 76, in init self.createIndex() File "coco-caption/pycocotools/coco.py", line 93, in createIndex if self.dataset['type'] == 'instances': KeyError: 'type'

yangxuntu commented 4 years ago

I do not meet this problem, it more likely a problem caused by the mismatch between somewhere and COCO package, you need to debug it by yourself.

发件人: JingJ_Liu notifications@github.com 发送时间: 2019年10月11日 9:21 收件人: yangxuntu/SGAE SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sg; Comment comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7)

What is your problem? … ____ 发件人: JingJ_Liu notifications@github.commailto:notifications@github.com 发送时间: 2019年10月11日 9:07 收件人: yangxuntu/SGAE SGAE@noreply.github.commailto:SGAE@noreply.github.com 抄送: #YANG XU# S170018@e.ntu.edu.sgmailto:S170018@e.ntu.edu.sg; Comment comment@noreply.github.commailto:comment@noreply.github.com 主题: Re: [yangxuntu/SGAE] How do you generate coco_pred_sg.zip and coco_spice_sg2.zip, and what does the value in those npy files means? (#7https://github.com/yangxuntu/SGAE/issues/7) Hi, thanks for you guidance for generating scene graphs for sentenceshttps://github.com/yangxuntu/SGAE#generating-scene-graphs. I've generated the sg.json file. Could you provide the script to generate from sg.json to coco_spice_sg2.zip? The demo file of sg.json for the above image with id #391895 is pasted here FYI. {"391895":{"subject":["back ","helmet ","man ","man ","man ","man ","man ","man ","man ","man ","foreground ","foreground ","motorcycle/bike ","road ","area ","path ","path ","person ","background "],"attribute":["node: back ","node: helmet ","attr: red ","node: shirt ","attr: red ","node: small ","node: man ","node: side ","attr: hill ","node: foreground ","node: motorcycle/bike ","attr: motor ","node: road ","attr: dirt ","node: area ","attr: verdant ","node: path ","attr: dirt ","node: mountain ","attr: cloud-wreathed ","node: bridge ","node: person ","attr: young ","node: hat ","attr: red ","node: background ","node: countryside "],"rela":["of ","on ","on ","ride ","ride on ","ride on ","mope on ","with ","in ","in ","of ","of ","on ","on ","with ","rest to ","with ","on ","of "],"object":["motorcycle/bike ","small ","motorcycle/bike ","motorcycle/bike ","back ","road ","road ","helmet ","shirt ","hat ","area ","background ","side ","countryside ","bridge ","foreground ","person ","motorcycle/bike ","mountain "]}} Hi，I see you have already use creat_coco_sg.py to generate sg.json .But I have some problem, Can you give me your Email?Thank you so much. D You are receiving this because you commented. Reply to this email directly, view it on GitHub<#7https://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUOQH6DU7CXO2AB3VRO3QN7GWXA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6NOAA#issuecomment-540858112>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJEJUOQH7S7ZJDO5V6JOACTQN7GWXANCNFSM4IEKOZSQ.

Thanks for your quick reply.The problem is: ''' loading annotations into memory... 0:00:03.230310 creating index... Traceback (most recent call last): File "create_coco_sg.py", line 19, in coco_train = COCO(train_path) File "coco-caption/pycocotools/coco.py", line 76, in init self.createIndex() File "coco-caption/pycocotools/coco.py", line 93, in createIndex if self.dataset['type'] == 'instances': KeyError: 'type'

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/yangxuntu/SGAE/issues/7?email_source=notifications&email_token=AJEJUORVYQHW2DDAMLFQ24TQN7ILPA5CNFSM4IEKOZS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6OB5Q#issuecomment-540860662, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJEJUOU4WISVR3BOERSOXGLQN7ILPANCNFSM4IEKOZSQ.

JingJLiu commented 4 years ago

Ok，thank you again~

forence commented 4 years ago

Try this file. …

Hi yang,

Thanks for your quick reply. I'd watched your lastest commit, but It seems we even don't have a way to extract sg.json file from scratch. create_coco_sg.py doesn't work.

You need to replace spice-1.0.jar in coco_caption/pycocoevalcap/spice with spice-1.0.jar in the main folder provided by the author, they are not the same.