Missing value-inputs in question generation

[JensNevens]

Hello

I have generated a series of images and questions using the scripts in this repository. For the question generation, I have only used the one hop template. I think that some of the functional programs in the output JSON file are missing some information.

Here is an example:

{"question": "There is a cube; what number of tiny green objects are right of it?",
 "program": [
    {
      "type": "scene",
      "inputs": [],
      "_output": [0, 1, 2, 3, 4, 5],
      "value_inputs": []
    },
    {
      "type": "filter_shape",
      "inputs": [0],
      "_output": [5],
      "value_inputs": []
    },
    {
      "type": "unique",
      "inputs": [1],
      "_output": 5,
      "value_inputs": []
    },
    {
      "type": "relate",
      "inputs": [2],
      "_output": [0, 1, 2, 3, 4],
      "value_inputs": ["right"]
    },
    {
      "type": "filter_size",
      "inputs": [3],
      "_output": [2],
      "value_inputs": ["small"]
    },
    {
      "type": "filter_color",
      "inputs": [4],
      "_output": [],
      "value_inputs": ["green"]
    },
    {
      "type": "count",
      "inputs": [5],
      "_output": 0,
      "value_inputs": []
    }
  ],

Shouldn't the first filter_shape function have ["cube"] as value_inputs?

This issue is not only restricted to shape, it also applies to size, color and material. Also, the issue is not limited to questions that start with There is a.... Here is a second example:

{"question": "Are there any large spheres on the left side of the tiny brown object?",
 "program": [
    {
      "type": "scene",
      "inputs": [],
      "_output": [0, 1, 2, 3, 4, 5],
      "value_inputs": []
    },
    {
      "type": "filter_size",
      "inputs": [0],
      "_output": [
        2,5],
      "value_inputs": []
    },
    {
      "type": "filter_color",
      "inputs": [1],
      "_output": [2],
      "value_inputs": []
    },
    {
      "type": "unique",
      "inputs": [2],
      "_output": 2,
      "value_inputs": []
    },
    {
      "type": "relate",
      "inputs": [3],
      "_output": [
        1,
        3,
        4,5],
      "value_inputs": ["left"]
    },
    {
      "type": "filter_size",
      "inputs": [4],
      "_output": [
        1,
        3,4],
      "value_inputs": ["large"]
    },
    {
      "type": "filter_shape",
      "inputs": [5],
      "_output": [
        3,4],
      "value_inputs": ["sphere"]
    },
    {
      "type": "exist",
      "inputs": [6],
      "_output": true,
      "value_inputs": []
    }
  ],

In this example, the first filter_size is missing ["tiny"] as value_inputs and the first filter_color is missing ["brown"] as value_inputs.

In both sentences, the filter functions later in the program do contain the correct value_inputs.

[JerryLingjieMei]

I have seen similar issues with question generation. My solution is to change the last lines in the main function to the following

  for q in questions:
        for f in q['program']:
            f['value_inputs'] = f['side_inputs'] if 'side_inputs' in f else []

Not sure why this would solve the issue though