search

rsummers11 / CADLab

Imaging Biomarkers and Computer-Aided Diagnosis Laboratory
https://www.cc.nih.gov/meet-our-doctors/rsummers.html
441 stars 187 forks source link

Issue with the training. #36

Closed cnu1439 closed 4 years ago

cnu1439 commented 4 years ago

Hi,

I was training body_part_regressor from scratch with DeepLesion dataset but the network is not converging. Is there any pre processing required for the dataset other than enabling "IMG_IS_16bit: True" in the config.yml. Attaching training log.

Thank You, Srinivas.

log.traintest.04-0621-49-35.txt

viggin commented 4 years ago

Hi Srinivas, I haven't used the caffe code for a while. I think you can manually check the data loader to make sure the preprocessing is working and the images can be visualized correctly. You can also try to train using one or two volumes first which should definitely converge easily.

cnu1439 commented 4 years ago

Hi viggin, Like you suggested. I have trained the model on 4 volumes, it is converging. And I have seen your list of volumes put for training in 'train_volume_list_example.txt', but 2 volumes repeated 10 times each(I also did the same thing with 4 volumes). Is it on purpose to have them repeated ? Thank You, Srinivas.

viggin commented 4 years ago

The repetition is a toy example and have no meanings. If 4 volumes can converge then more volumes should also do. I remember I trained on DeepLesion and it can converge.