search

ljwztc / CLIP-Driven-Universal-Model

[ICCV 2023] CLIP-Driven Universal Model; Rank first in MSD Competition.
Other
521 stars 58 forks source link

Training about convergence problem #37

Closed wk100869 closed 9 months ago

wk100869 commented 9 months ago

Hello, may I ask how many epochs you have trained and the dice loss value at the final convergence

When I trained using three datasets (liver, kidney, and tumor), 04-LITS, 05KITS, and 10-3liver, I trained for 200 epochs. When finally converged, the dice loss value was around 0.7，I think it's very high. However, in the test set, the dice value range for liver and kidney was around 0.96, and the dice value for tumor was around 0.65. I don't understand why the dice loss value was so high during training convergence. According to conventional understanding, when training convergence, dice predicted (1-0.7=) 0.3 correctly, The test results should also be very low, but the test set results are normal. Could you please answer them? Thank you very much

ljwztc commented 9 months ago

Yes, the dice loss was around roughly 0.65. I guess that when calculating the Dice loss, probabilities are used, which limits the loss to decrease to only 0.6.

sharonlee12 commented 6 months ago

Hello, may I ask how many epochs you have trained and the dice loss value at the final convergence

When I trained using three datasets (liver, kidney, and tumor), 04-LITS, 05KITS, and 10-3liver, I trained for 200 epochs. When finally converged, the dice loss value was around 0.7，I think it's very high. However, in the test set, the dice value range for liver and kidney was around 0.96, and the dice value for tumor was around 0.65. I don't understand why the dice loss value was so high during training convergence. According to conventional understanding, when training convergence, dice predicted (1-0.7=) 0.3 correctly, The test results should also be very low, but the test set results are normal. Could you please answer them? Thank you very much

Hello,I am sorry to bother you,When I train BTCV, the diceloss is always around 0.98. Even up to 2000 epochs, this is the same. Have you ever encountered a similar problem

sharonlee12 commented 6 months ago

Hello,I am sorry to bother you,I am training on 04-LITS for about 500epoch,my loss decreased and the dice loss value was around 0.7 just like you,but in the test set, the dice value for liver was all 0,this is very confusing,could I have a look at your training code?Since I used the checkpoint provided by the author to achieve a result of around 0.96, I think it may be a problem when I was training,Looking forward to your reply, thank you!