How did you create your json annotations 65/15, vocabulary, and word-to-vec ?

[harrytrinh96]

Could you please let me know how did you create datasets? I mean:

1. what are vocabulary_w2v.txt, word_w2v_withbg_48_17.txt, word_w2v_withbg_65_15.txt ? And how did you create them?

2. Also, How did you create files in annotations folders such as word_w2v_withbg_65_15.txt, instances_val2014_gzsi_65_15.txt, and instances_val2014_unseen_65_15.json ? Thank you.

[zhengye1995]

I created the dataset following two papers:

1. Shafin Rahman, Salman Khan, and Nick Barnes. "Improved Visual-Semantic Alignment for Zero-Shot Object Detection," 34th AAAI Conference on Artificial Intelligence, (AAAI), New York, US, 2020 (splits MS-COCO to 65 seen and 15 unseen classes)
2. https://arxiv.org/abs/1804.04340 (splits MS-COCO to 48 seen and 17 unseen classes)

I used the word2vec from https://github.com/salman-h-khan/PL-ZSD_Release.

word_w2v_withbg_48_17.txt and word_w2v_withbg_65_15.txt are the word2vecs of seen and unseen classes in two splits. "withbg" means with background word2vec for "background" class.

How to generate the label files like instances_val2014_unseen_65_15.json:

1. For training label files : a. Load the training label file instances_train2014.json (provided by the official MS-COCO) b. Remove all images, all annotations, all categories belong to unseen classes and just keep seen classes data c. Save the filtered label file (instances_train2014_seen_48_17.json for 48/17 split and instances_train2014_seen_65_15.json for 65/15 split)

2. For testing label file: a. for zsi task (only needs unseen classes):

    i. Load the val label file  instances_val2014.json (provided by the official MS-COCO)
    ii. Remove all annotations, all categories belong to seen classes (here we just use the test image lists from above paper)
   iii. Save the filtered label file (instances_val2014_unseen_48_17.json for 48/17 split and instances_val2014_unseen_65_15.json for 65/15 split)

   b. for gzsi task (needs both seen and unseen classes):

   i. Load the val label file  instances_val2014.json (provided by the official MS-COCO)
   ii. Keep all seen and unseen classes (here we just use the test image lists from above paper)
   iii. Save the filtered label file (instances_val2014_gzsi_48_17.json for 48/17 split and instances_val2014_gzsi_65_15.json for 65/15 split)

[harrytrinh96]

Thank you very much for your answers. I will look into the papers you provided. If i don't have further questions for this topic, I will close this issue myself.

[algoteam5]

@zhengye1995 Could you please share your code to load instances_val2014_unseen_65_15.json: which can "remove all images, all annotations, all categories belong to unseen classes and just keep seen classes data"?

[Na-Z]

@zhengye1995 May I know how u generate vectors for classes with multiple words (e.g. 'traffic light' in COCO)?