Regarding zero shot evaluation

[galsk87]

❓ Questions and Help

Hi, First of all this is a great repository. I enjoyed reading your paper.

I wanted to know regarding the zero shot evaluation, what are the splits you used because when i checked all the triplets in the training set and test set that are most commonly used (Language priors, iterative message passing and motifs) i did not find any triplets that appear in the test set and did not appear in the train set.

[KaihuaTang]

I used the common 150(objects)-50(predicates) splits the same as the Neural-MOTIFS. The potential maximum triplet combinations are 150*50*150, and I neglect all the combinations appeared in the training set. All valid zero-shot combinations are stored in https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/evaluation/vg/zeroshot_triplet.pytorch

The valid combinations in the training set are calculated by running https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/visual_genome.py for one epoch.

[galsk87]

So if I understand correctly: We have 15015050 valid triplets. In the training set there is around 12k triplets. So the rest of the triplets are valid as zero-shot triplets?

בתאריך יום ב׳, 13 באפר׳ 2020, 17:15, מאת Kaihua Tang ‏< notifications@github.com>:

I used the common 150(objects)-50(predicates) splits the same as the Neural-MOTIFS. The potential maximum triplet combinations are 15050150, and I neglect all the combinations appeared in the training set. All valid zero-shot combinations are stored in https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/evaluation/vg/zeroshot_triplet.pytorch

The valid combinations in the training set are calculated by running https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/visual_genome.py for one epoch.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/issues/20#issuecomment-612916498, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZQGMW667MO3OKHY7WLRQLRMMNBNANCNFSM4MGKQGLQ .

[KaihuaTang]

So if I understand correctly: We have 15015050 valid triplets. In the training set there is around 12k triplets. So the rest of the triplets are valid as zero-shot triplets? בתאריך יום ב׳, 13 באפר׳ 2020, 17:15, מאת Kaihua Tang ‏< notifications@github.com>: … I used the common 150(objects)-50(predicates) splits the same as the Neural-MOTIFS. The potential maximum triplet combinations are 15050150, and I neglect all the combinations appeared in the training set. All valid zero-shot combinations are stored in https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/evaluation/vg/zeroshot_triplet.pytorch The valid combinations in the training set are calculated by running https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/maskrcnn_benchmark/data/datasets/visual_genome.py for one epoch. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZQGMW667MO3OKHY7WLRQLRMMNBNANCNFSM4MGKQGLQ .

yes

[galsk87]

Ok so i tested this premise: I iterated over the training relationships and saved all the triplet combinations in the training set. I got that the array in 'zeroshot_triplet.pytorch' is a subset of the training set triplets/

So again I don't understand how the calculation is done?