Bug report: the model does not use any relationship in training.

[zhiyong1997]

Hi, I possibly have found a major bug in the data loading code sg2im/data/vg.py, which hides all the relationship and the model only sees the type and number of objects.

In sg2im/data/vg.py line 66, the getitem function use a python set to store the object index that is involved in some relationships, keep a certain number of objects, and then keep the relationships whose subjects and objects are both kept. But the following simple example shows that this doesn't work.

s = set()
x = torch.LongTensor([1, 2, 3])

s.add(x[0])
x[0] in s  # False

When adding

assert len(triples) == 0

to line 134, vg.py, the training can go through, which proves that the model does not see any relationship except for __in_image__

When generating images with the pre-trained model vg124.pt, the following two scene graphs generates almost the same images.

{
      "objects": ["sky", "grass", "sheep", "sheep", "tree", "ocean", "boat"],
      "relationships": [
        [0, "above", 1],
        [2, "standing on", 1],
        [3, "by", 2],
        [4, "behind", 2],
        [5, "by", 4],
        [6, "on", 1]
      ]   
}
{
      "objects": ["sky", "grass", "sheep", "sheep", "tree", "ocean", "boat"],
      "relationships": [
        [0, "standing on", 1],
        [2, "standing on", 1],
        [3, "standing on", 2],
        [4, "standing on", 2],
        [5, "standing on", 4],
        [6, "standing on", 1]
      ] 
}

I recommend to turn pytorch scalar tensor to python int object before put it in python set, and the pre-trained model may need some update.

Thanks

[jcjohnson]

Thanks for the detailed bug report!

You are 100% correct, this is a bug, and should now be fixed in https://github.com/google/sg2im/commit/57e1c087b0b7c519b20c67c13c5063675c7a91d8.

Prior to PyTorch 0.4, there were no PyTorch scalars so any indexing operation that would return a scalar just returned a Python scalar. All of the pretrained models were trained with PyTorch 0.3.0 so they were trained with proper data loading and should not be affected by this.

[zhiyong1997]

Thanks for reply. I think that solves the problem.