Different model output for training for different runs for M4C on ST-VQA dataset

[soonchangAI]

❓ Questions and Help

Dataset: ST-VQA, Model: M4C Hi, I noticed the model output (of same example) for training differs for different runs. For prediction, the model output (of same example) for different runs are the same. I fixed the answers selected as idx_seq with np.random.seed(0) in processors.py. What causes this behavior?

For example: Training: Different model output first run:

'scores': tensor([[[-1.9008e+01, -9.0379e+00, -1.2226e+01,  ..., -1.0017e+04,
          -1.0018e+04, -1.0018e+04],
         [-1.9977e+01, -2.1508e+01, -5.9142e+00,  ..., -1.0014e+04,
          -1.0015e+04, -1.0014e+04],
         [-2.4891e+01, -2.0198e+01,  1.0535e+01,  ..., -1.0022e+04,
          -1.0022e+04, -1.0022e+04],
         ...,

second run:

'scores': tensor([[[-2.0432e+01, -1.1464e+01, -1.2194e+01,  ..., -1.0019e+04,
          -1.0020e+04, -1.0019e+04],
         [-1.9976e+01, -2.1802e+01, -5.6499e+00,  ..., -1.0014e+04,
          -1.0014e+04, -1.0014e+04],
         [-2.4550e+01, -2.0520e+01,  1.0426e+01,  ..., -1.0021e+04,
          -1.0021e+04, -1.0021e+04],
         ...,

Prediction: Same model output first run:

'scores': tensor([[[-2.0423e+01, -1.1363e+01, -1.1639e+01,  ..., -1.0020e+04,
          -1.0020e+04, -1.0020e+04],
         [-2.0124e+01, -2.1779e+01, -5.5835e+00,  ..., -1.0014e+04,
          -1.0014e+04, -1.0014e+04],
         [-2.4863e+01, -2.1161e+01,  1.0917e+01,  ..., -1.0023e+04,
          -1.0023e+04, -1.0023e+04],
         ...,

second run:

'scores': tensor([[[-2.0423e+01, -1.1363e+01, -1.1639e+01,  ..., -1.0020e+04,
          -1.0020e+04, -1.0020e+04],
         [-2.0124e+01, -2.1779e+01, -5.5835e+00,  ..., -1.0014e+04,
          -1.0014e+04, -1.0014e+04],
         [-2.4863e+01, -2.1161e+01,  1.0917e+01,  ..., -1.0023e+04,
          -1.0023e+04, -1.0023e+04],
         ...,

[soonchangAI]

Dropout is causing this. Set a seed by torch.manual_seed(0) at forward function of the model to get same output