Unable to Reproduce 12 Epoch Results in the Paper

[yeliudev]

Hi @Jingkang50, thanks for your great work. I ran a 12-epoch training for PSGFormer by modifying the config from 40/60 epochs to 8/12 epochs, and setting eval_pan_rels=False according to your suggestion. Below are the evaluation results at the 12th epoch, which are largely different from the 12th epoch results in Table 2 in your paper. May I know if there are any further modifications to reproduce the results? Thanks.

====================================================================================================
SGG eval:  R @ 20: 0.0941;  R @ 50: 0.1138;  R @ 100: 0.1159;  for mode=sgdet, type=Recall.
SGG eval:  R @ 20: 0.1778;  R @ 50: 0.2223;  R @ 100: 0.2278;  for mode=phrdet, type=Recall.
SGG eval:  R @ 20: nan;  R @ 50: nan;  R @ 100: nan;  for mode=sgdet, type=NoGraphConstraint @ 56 Recall.
SGG eval:  R @ 20: nan;  R @ 50: nan;  R @ 100: nan;  for mode=phrdet, type=NoGraphConstraint @ 56 Recall.
SGG eval:  mR @ 20: 0.0584;  mR @ 50: 0.0785;  mR @ 100: 0.0794;  for mode=sgdet, type=Mean Recall.
over    in front of beside  on  in  attached to hanging from    on back of  falling off going down  painted on  walking on  running on  crossing    standing on lying on    sitting on  flying over jumping over    jumping from    wearing holding carrying    looking at  guiding kissing eating  drinking    feeding biting  catching    picking playing with    chasing climbing    cleaning    playing touching    pushing pulling opening cooking talking to  throwing    slicing driving riding  parked on   driving on  about to hit    kicking swinging    entering    exiting enclosing   leaning on
0.3282  0.0558  0.0445  0.0521  0.0708  0.0394  0.1034  0.0000  0.0000  0.0000  0.0000  0.1905  0.1008  0.0000  0.1953  0.0399  0.0889  0.1840  0.0000  0.0000  0.2012  0.2540  0.0556  0.1662  0.0000  0.0000  0.1023  0.0000  0.0000  0.1667  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.2200  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.1628  0.3581  0.1288  0.2737  0.5926  0.2500  0.0000  0.0000  0.0000  0.0219  0.0000
+--------------+--------+--------------+--------+--------------+--------+
| predicate    | Rec100 | predicate    | Rec100 | predicate    | Rec100 |
+--------------+--------+--------------+--------+--------------+--------+
| over         | 0.3282 | in front of  | 0.0558 | beside       | 0.0445 |
| on           | 0.0521 | in           | 0.0708 | attached to  | 0.0394 |
| hanging from | 0.1034 | on back of   | 0.0000 | falling off  | 0.0000 |
| going down   | 0.0000 | painted on   | 0.0000 | walking on   | 0.1905 |
| running on   | 0.1008 | crossing     | 0.0000 | standing on  | 0.1953 |
| lying on     | 0.0399 | sitting on   | 0.0889 | flying over  | 0.1840 |
| jumping over | 0.0000 | jumping from | 0.0000 | wearing      | 0.2012 |
| holding      | 0.2540 | carrying     | 0.0556 | looking at   | 0.1662 |
| guiding      | 0.0000 | kissing      | 0.0000 | eating       | 0.1023 |
| drinking     | 0.0000 | feeding      | 0.0000 | biting       | 0.1667 |
| catching     | 0.0000 | picking      | 0.0000 | playing with | 0.0000 |
| chasing      | 0.0000 | climbing     | 0.0000 | cleaning     | 0.0000 |
| playing      | 0.2200 | touching     | 0.0000 | pushing      | 0.0000 |
| pulling      | 0.0000 | opening      | 0.0000 | cooking      | 0.0000 |
| talking to   | 0.0000 | throwing     | 0.0000 | slicing      | 0.0000 |
| driving      | 0.1628 | riding       | 0.3581 | parked on    | 0.1288 |
| driving on   | 0.2737 | about to hit | 0.5926 | kicking      | 0.2500 |
| swinging     | 0.0000 | entering     | 0.0000 | exiting      | 0.0000 |
| enclosing    | 0.0219 | leaning on   | 0.0000 | None         | None   |
+--------------+--------+--------------+--------+--------------+--------+
SGG eval:  mR @ 20: 0.1017;  mR @ 50: 0.1474;  mR @ 100: 0.1502;  for mode=phrdet, type=Mean Recall.
SGG eval:  mR @ 20: 0.0000;  mR @ 50: 0.0000;  mR @ 100: 0.0000;  for mode=sgdet, type=NoGraphConstraint @ 56 Mean Recall.
SGG eval:  mR @ 20: 0.0000;  mR @ 50: 0.0000;  mR @ 100: 0.0000;  for mode=phrdet, type=NoGraphConstraint @ 56 Mean Recall.
====================================================================================================

[Jingkang50]

Thank you for your interest in our work. Maybe check out here https://github.com/Jingkang50/OpenPSG/issues/33 for log reference. Some potential issues could be different configs (num_gpu and lr), and the loading of pretrained models.

[yeliudev]

Thanks for your reply. I've changed the base lr from 1e-3 to 1e-4 (the paper said PSGTR uses 1e-4, but the config of PSGFormer uses 1e-3, not sure whether it is correct), and the results became R@20: 0.1550; R@50: 0.1829; R@100: 0.1845, which are still worse than the reported ones.

[yeliudev]

Comparing my training log and your uploaded ones, I notice that several losses and the HTriMatcher are modified/removed in your released config. Do these modifications affect the performances?

Left: The released config & Right: The uploaded training log

[yeliudev]

@Jingkang50 The 60 epochs schedule can not reproduce the results as well. Could you please share the codebase used for obtaining the checkpoints? It seems that the code used for obtaining checkpoints and the current release one are different. Thanks!

[Jingkang50]

The released codebase is just the one that we get the final score (as a few participants from PSG challenges was able to get the corresponding results). Are you using the config here? It seems the loss weight aligned with the Left: The released config. For the difference in the config, we delete/change the lines either because some losses are unused or they don't matter much.