koriavinash1 / DigitalHistoPath

MIT License
23 stars 9 forks source link

strange behavior in model training #21

Open heba9004 opened 3 years ago

heba9004 commented 3 years ago

Hello, I need your help and experience, I would like to ask about something strange occurred when I tried a variant of Unet on your same framework, as the epochs increased the performance in terms of Jaccard index become worse (not improving as it supposed). please see below some results:

epoch 3 Jaccard for Training_phase_1_001 = 0.278841 Jaccard for Training_phase_1_002 = 0.060954 epoch 2 Jaccard for Training_phase_1_001 = 0.286575 Jaccard for Training_phase_1_002 = 0.084818 epoch 1 Jaccard for Training_phase_1_001 = 0.350196 Jaccard for Training_phase_1_002 = 0.170044

Is my learning rate low? or does I miss something somewhere?

one more thing, is it normal that training one epoch take 3.5 hours to complete using GPU?

Thank you for your help and code as well.