facebookresearch / clevr-dataset-gen

A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Other
577 stars 204 forks source link

Zeroeth template in compare_integer.json can never return true #37

Open ikb-a opened 1 year ago

ikb-a commented 1 year ago

First and foremost, it goes without saying, but great repo and dataset!

Moving on to the issue, I believe there's a slight error in the zeroeth template of compare_integer.json

Specifically, the zeroeth template (with indenting for legibility) is as follows:

    "constraints": [
      {
        "params": [
          1,
          3
        ],
        "type": "OUT_NEQ"
      }
    ],
    "nodes": [
      {
        "inputs": [],
        "type": "scene"
      },
      {
        "inputs": [
          0
        ],
        "side_inputs": [
          "<Z>",
          "<C>",
          "<M>",
          "<S>"
        ],
        "type": "filter_count"
      },
      {
        "inputs": [],
        "type": "scene"
      },
      {
        "inputs": [
          2
        ],
        "side_inputs": [
          "<Z2>",
          "<C2>",
          "<M2>",
          "<S2>"
        ],
        "type": "filter_count"
      },
      {
        "inputs": [
          1,
          3
        ],
        "type": "equal_integer"
      }
    ],
    "params": [
      {
        "name": "<Z>",
        "type": "Size"
      },
      {
        "name": "<C>",
        "type": "Color"
      },
      {
        "name": "<M>",
        "type": "Material"
      },
      {
        "name": "<S>",
        "type": "Shape"
      },
      {
        "name": "<Z2>",
        "type": "Size"
      },
      {
        "name": "<C2>",
        "type": "Color"
      },
      {
        "name": "<M2>",
        "type": "Material"
      },
      {
        "name": "<S2>",
        "type": "Shape"
      }
    ],
    "text": [
      "Are there an equal number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
      "Are there the same number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
      "Is the number of <Z> <C> <M> <S>s the same as the number of <Z2> <C2> <M2> <S2>s?"
    ]
  },

Note the constraint is OUT_NEQ between nodes 1 and 3. However, nodes 1 and 3 are filter_count nodes rather than filter nodes, so the constraint is that the counts (rather than the objects found) must differ. Consequently, the answer is always false (there cannot be an equal number of objects, because our constraint forces the object count to differ). I threw together a quick script to try and check the answers in the CLEVR training set, and I believe it supports my conclusions.

It's not a serious issue, but it confused me and took me a while to figure out what was going on -- I hope this spares someone else some debugging time. Cheers!