GNN decode doesn't work correctly when lacking strokes for a scene

[spigo900]

The GNN decode script crashes in single-object mode if any scene is missing strokes. This is because it doesn't know how to align the model outputs with the situations when some situations have no outputs. (It looks like the decode script also shouldn't work in multi-object when some situations lack strokes, though it may not crash.)

To fix this, we need to keep track of the situation number ->* stroke data relationship when loading stroke data. An easy way to do this would be to track a situation_number ->* input_index mapping in get_stroke_data() and add this to the return value, then use that in the decode script to build predictions_by_object. We could then get rid of the outputs.size(0) == curriculum_params["num_dirs"] assertion, instead checking the "we have enough outputs" condition using zip_equal() in the prediction update code. As a bonus I think this could also remove the need for special logic to save multi-object decodes.

[boyleconnor]

I noticed this bug as well when I ran the GNN scripts on the output of the stroke extraction script. I wasn't sure whether this was a bug, but I noticed that all of the situations in the already pre-processed m5 curriculum had at least one object in them (as recognized by the stroke extraction script), so I think I assumed I was missing something I would understand later.

I am curious, how did the creators of the m5_objects_v0_with_mugs curriculum ensure that each situation would have at least one recognized object?

[spigo900]

@boyleconnor About m5_objects_v0_with_mugs, basically we had gotten pre-baked stroke extractions and the curriculum was in a different layout. So when we reformatted it to the expected layout, we just dropped any situations that were missing strokes (see scripts/reorganize_curriculum.py).