search

Tushar-N / attributes-as-operators

Attribute-Object Visual Composition using Attributes as Operators
MIT License
64 stars 17 forks source link

Evaluation Metric #2

Closed TsinghuaPhantom closed 6 years ago

TsinghuaPhantom commented 6 years ago

Hi, i have a question in your evaluation table. The chance method in open world. Shouldn't the accuracy be around 1/(115*245) which is 0.0035%, can you correct me if I understand wrong? Thank you.

Tushar-N commented 6 years ago

The dataset doesn't have every object paired with every attribute. Some compositions are less meaningful (like "rotten television") and don't have images in the dataset. The total number of compositions is around 1.9k, which are the pairs we evaluate over in the open world (vs. the 700 in the closed world). You are right in a sense though - we bumped up our search space from 700 to 1.9k, but the true set of compositions (visually grounded or not) is 115*245, however, we did not report those numbers in the paper.

TsinghuaPhantom commented 6 years ago

Ah, I see. Thank you very much!