First and foremost, it goes without saying, but great repo and dataset!
Moving on to the issue, I believe there's a slight error in the zeroeth template of compare_integer.json
Specifically, the zeroeth template (with indenting for legibility) is as follows:
"constraints": [
{
"params": [
1,
3
],
"type": "OUT_NEQ"
}
],
"nodes": [
{
"inputs": [],
"type": "scene"
},
{
"inputs": [
0
],
"side_inputs": [
"<Z>",
"<C>",
"<M>",
"<S>"
],
"type": "filter_count"
},
{
"inputs": [],
"type": "scene"
},
{
"inputs": [
2
],
"side_inputs": [
"<Z2>",
"<C2>",
"<M2>",
"<S2>"
],
"type": "filter_count"
},
{
"inputs": [
1,
3
],
"type": "equal_integer"
}
],
"params": [
{
"name": "<Z>",
"type": "Size"
},
{
"name": "<C>",
"type": "Color"
},
{
"name": "<M>",
"type": "Material"
},
{
"name": "<S>",
"type": "Shape"
},
{
"name": "<Z2>",
"type": "Size"
},
{
"name": "<C2>",
"type": "Color"
},
{
"name": "<M2>",
"type": "Material"
},
{
"name": "<S2>",
"type": "Shape"
}
],
"text": [
"Are there an equal number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
"Are there the same number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
"Is the number of <Z> <C> <M> <S>s the same as the number of <Z2> <C2> <M2> <S2>s?"
]
},
Note the constraint is OUT_NEQ between nodes 1 and 3. However, nodes 1 and 3 are filter_count nodes rather than filter nodes, so the constraint is that the counts (rather than the objects found) must differ. Consequently, the answer is always false (there cannot be an equal number of objects, because our constraint forces the object count to differ). I threw together a quick script to try and check the answers in the CLEVR training set, and I believe it supports my conclusions.
It's not a serious issue, but it confused me and took me a while to figure out what was going on -- I hope this spares someone else some debugging time. Cheers!
First and foremost, it goes without saying, but great repo and dataset!
Moving on to the issue, I believe there's a slight error in the zeroeth template of compare_integer.json
Specifically, the zeroeth template (with indenting for legibility) is as follows:
Note the constraint is OUT_NEQ between nodes 1 and 3. However, nodes 1 and 3 are
filter_count
nodes rather than filter nodes, so the constraint is that the counts (rather than the objects found) must differ. Consequently, the answer is always false (there cannot be an equal number of objects, because our constraint forces the object count to differ). I threw together a quick script to try and check the answers in the CLEVR training set, and I believe it supports my conclusions.It's not a serious issue, but it confused me and took me a while to figure out what was going on -- I hope this spares someone else some debugging time. Cheers!