search

YiwuZhong / SGG_from_NLS

[ICCV 2021] Official code for "Learning to Generate Scene Graph from Natural Language Supervision"
Other
100 stars 10 forks source link

Would you release the code for the pseudo label generation and assignment? #4

Open Scarecrow0 opened 2 years ago

Scarecrow0 commented 2 years ago

Hi!

Thanks for your interesting work and nice codebase on weakly supervised SGG. Your high-quality codebase and comments make me understand your proposed method concretely.

The pseude label generation and assignment is a much different design from previous works of weakly SGG. The generated labels have been provided here. However, I'm wondering about the technical details of seudo label generation and assignment in your work. Do you have a plan to release the preprocess code for the pseudo label generation?

I'm looking forward to your reply.

JingyuLi-code commented 2 years ago

Hi, Yiwu. Same question! In my understanding, the quality of pseudo labels may greatly affect the quality of scene graph generation. Would you release the code for the pseudo label generation and assignment? Thanks!

zyong812 commented 2 years ago

Same question +1 about parsing labels from language. Have not found related code.

YiwuZhong commented 2 years ago

Hi @Scarecrow0 @JingyuLi-code @zyong812, thanks for your interests in our work!

Sorry for late response. I haven't got a chance to fully clean my preprocessing code. As you requested, I uploaded my functions used for preprocessing (https://github.com/YiwuZhong/SGG_from_NLS/tree/main/preprocess#pseudo-label-creation). You could use them as reference for your case.

JingyuLi-code commented 2 years ago

Hi @Scarecrow0 @JingyuLi-code @zyong812, thanks for your interests in our work!

Sorry for late response. I haven't got a chance to fully clean my preprocessing code. As you requested, I uploaded my functions used for preprocessing (https://github.com/YiwuZhong/SGG_from_NLS/tree/main/preprocess#pseudo-label-creation). You could use them as reference for your case.

Hi, Yiwu. Thanks for your reply. I have checked the link_cap_vg.py, which converts the predicted caption categories into VG standard categories.

How about matching sentence triplets to pairs of regions provided by the object detector. as the paper displays 'This is done by a greedy matching between every triplet from the caption and each region pair from the image.' How do I generate COCO_triplet_labels.npy files?

For pseudo label assignment, many image region pairs may correspond to a caption triple, this may lead to a wrong pseudo label. For example, for image region pairs 'man0', 'man1', 'boat', this will have <man0, boat>, <man1, boat> to correspond to a same caption triple <man, boat>. However, <man1, boat> may have no semantic relation. Do you have a good way to solve this problem?

YiwuZhong commented 2 years ago

Hi, Yiwu. Thanks for your reply. I have checked the link_cap_vg.py, which converts the predicted caption categories into VG standard categories.

How about matching sentence triplets to pairs of regions provided by the object detector. as the paper displays 'This is done by a greedy matching between every triplet from the caption and each region pair from the image.' How do I generate COCO_triplet_labels.npy files?

For pseudo label assignment, many image region pairs may correspond to a caption triple, this may lead to a wrong pseudo label. For example, for image region pairs 'man0', 'man1', 'boat', this will have <man0, boat>, <man1, boat> to correspond to a same caption triple <man, boat>. However, <man1, boat> may have no semantic relation. Do you have a good way to solve this problem?

@JingyuLi-code Thanks for your interest in our work. I provided the functions I used to process the pseudo labels link_det_cap.py.

The key idea is simply using WordNet or something equivalent to match the detection labels and caption nouns. If there are multiple objects with same detection class, we select the ones using heuristic approach. For example, inspired by MotifNet, we only care about the objects that have overlapping regions.

ZHUXUHAN commented 2 years ago

The key idea is simply using WordNet or something equivalent to match the detection labels and caption nouns. If there are multiple objects with same detection class, we select the ones using heuristic approach. For example, inspired by MotifNet, we only care about the objects that have overlapping regions.

Thank you for your work, but for this generation of pseudo labels, can you open source the relevant code, just for the generation of COCO_triplet_labels.npy.

shwetabhardwaj44 commented 1 year ago

Hi Authors, thanks for the code repository. I also have the same question: can you make the code for COCO_triplet_labels.npy available to generate pseudo labels for given COCO-captions dataset.

YiwuZhong commented 1 year ago

Just release the code for parsing triplets from captions (parse_captions.py): https://github.com/YiwuZhong/SGG_from_NLS/tree/main/preprocess#pseudo-label-creation