Training loss keeps going up

[Valeyards]

Hi @cwmok , I'm trying to train LapIRN in l2r 2020 task2 i.e., the inhale-exhale lung registration. I'm uisng Train_LapIRN_disp.py. During training procedure, the initial loss looks like

After some iterations it becomes...

The lr is default 1e-4 since I found if I change it to 1e-3 the situation is even worse.

I only did some minor changes on Dataset_Epoch in Functions.py and adjust the input shape to (192, 192, 208) as it is the original size right?

I'm using torch 1.7.1+cuda 11.0 on a single RTX 3090 Do you have any idea about this?

[cwmok]

1. We trained our model at half-resolution, i.e., 96x96x104.
2. We applied CT windowing - [100, 1518] to both scans.
3. You should define the region of interest (ROI) of the registration using masks (Cost function masking). For masking, please refer to the "Masks" section in https://simpleelastix.readthedocs.io/Introduction.html.
4. We found that NGF is more stable than NCC in this task. Yet, NCC yields slightly better performance when it works.

Other minor changes are summarized in the ppt slides below: Learn2Reg2021_Tony_oral.pdf

[Valeyards]

1. We trained our model at half-resolution, i.e., 96x96x104.
2. We applied CT windowing - [100, 1518] to both scans.
3. You should define the region of interest (ROI) of the registration using masks (Cost function masking). For masking, please refer to the "Masks" section in https://simpleelastix.readthedocs.io/Introduction.html.
4. We found that NGF is more stable than NCC in this task. Yet, NCC yields slightly better performance when it works.

Other minor changes are summarized in the ppt slides below: Learn2Reg2021_Tony_oral.pdf

Thank you!