search

tuyunbin / SCORER

[ICCV 2023] This is the Pytorch code for our paper "Self-Supervised Cross-View Representation Reconstruction for Change Captioning".
16 stars 1 forks source link

The code is not optimizing the generator #3

Closed matthewdm0816 closed 9 months ago

matthewdm0816 commented 9 months ago

Hi, I found in the train.py, the generator is not being optimized, and this leads to non-decreasing cycle loss. Is this a bug or an intended design? https://github.com/tuyunbin/SCORER/blob/2c12e34199508558c8eade6fe1a41da7c1797e98/train.py#L114-L115

tuyunbin commented 9 months ago

Hi, I did not include it in the optimizer, so the drop of cycle loss is not obvious. In my opinion, we only need to make the composite representation of caption and before-image close to the after-image representation in a certain range during overall training process. It is unnecessary to enforce the composite representation to be totally identical to the after-image representation (i.e., cycle loss close to zero), because the change caption not only includes visual difference information, but also refers to other information such as referents and non-visual words.

You can certainly try to optimize these parameters simultaneously, which also obtains the similar performance, where the cycle loss has an obvious drop (close to zero).

matthewdm0816 commented 9 months ago

thank you for your explanation!