Difficulty in Reproducing EgoNLQ Task Results

[Haawron]

Hello and thank you for your outstanding work on this project!

I am reaching out because I've encountered an issue while trying to reproduce the results for the EgoNLQ task. Specifically, I achieved an mIoU@0.3(R@1) score of 12.00, which slightly deviates from the reported score of 12.95. Upon reviewing the code, I noticed that the seeding process, which might be crucial for replicating the results, appears to be commented out (as seen in this line).

To better understand this discrepancy, could you possibly share the results obtained with a fixed seed, or provide guidance on how to replicate the reported scores more accurately?

Thank you in advance for your assistance and for the valuable contributions your work offers to the community.

Reported
+Epoch 12, Step 4236--+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  Rank@1  |  Rank@1  |   Rank@1  |  Rank@3  |  Rank@3  |   Rank@3  |  Rank@5  |  Rank@5  |   Rank@5  | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 |      |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  12.95   |   7.91   |   28.94   |  18.53   |  12.96   |   42.50   |  23.80   |  16.11   |   48.85   | 9.09 |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+

Reproduced (w/o seed fixed)
+Epoch 15, Step 5295--+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  Rank@1  |  Rank@1  |   Rank@1  |  Rank@3  |  Rank@3  |   Rank@3  |  Rank@5  |  Rank@5  |   Rank@5  | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 |      |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  12.00   |   7.59   |   26.36   |  17.63   |  12.34   |   40.66   |  19.93   |  14.46   |   47.57   | 8.72 |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+

Reproduced (w/ seed fixed: 12345)
+Epoch 14, Step 4752--+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  Rank@1  |  Rank@1  |   Rank@1  |  Rank@3  |  Rank@3  |   Rank@3  |  Rank@5  |  Rank@5  |   Rank@5  | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 | mIoU@0.3 | mIoU@0.5 | mIoU@0.01 |      |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+
|  11.74   |   7.36   |   24.57   |  16.88   |  11.56   |   39.73   |  20.19   |  14.22   |   46.08   | 8.44 |
+----------+----------+-----------+----------+----------+-----------+----------+----------+-----------+------+

Best regards, Hyogun Lee

[ShramanPramanick]

Hi Hyogun,

Thanks for raising the issue. During head-tuning the VSLNet, we did not set a seed. We have provided our NLQ head-tuning log here. Are you using the pre-extracted features provided here, or do you pre-train on your own? Depending on the GPU accelerator and the CUDA version you are using, deviation in results in decimal points can happen during training.

[Haawron]

I could get your result by running the code a few more times with the features you provided. I got 12.67 with setting the seed, which means the method is not that sensitive by seeds. Thank you for the great work!