Closed Blcony closed 6 years ago
simonmeister
commented
6 years ago No problem and you're welcome! Did you try it with the simplest loss (i.e. just census + smoothness without fb and occ handling)? Enabling occlusion handling without any pre-training done first without occlusion handling could prevent convergence. Also, what is the evaluation performance?
Blcony
commented
6 years ago Thanks for your reply! First, I want to use your network on my own dataset which contains many medical images in next step, and there isn't any groundtruth for my dataset. So when I replicate your network on Synthia, I just ignored the evaluation process, as a result there isn't any evaluation performance which can be provided. I'm so sorry about that. Second, I haven't tried the simplest loss for the train process on Synthia, because I don't have the plan to pretrain the mode firstly on my own dataset. Maybe I'll try the simplest loss on Synthia in next stage. So your opinion is that use the complicate losses directly could prevent convergence, and is there any other possibility? Third, do you advice me to pretrain the model with the simplest loss on my own dataset firstly, then add fb and occ to the loss when finetuneing on pretrained model? But my dataset just has one type of images, and the amount of the images aren't very big. Do you have any ideas about that? Thanks for your patient answer again, it means a lot for me. Thank you!
simonmeister
commented
6 years ago In any case, i would first suggest trying your training with the simplest loss . If that works better, you can try to enable the other loss terms after a few iterations (e.g. 100K or 200K). It shouldn't be an issue to use the same dataset for both stages. Did you visually inspect the predicted color coded flow field?
Hi, Simon! Sorry to bother you again. This time, I got some problems about loss curves from my own trained model on Synthia dataset. I wish that you can give me some advices, thank you very much, In details, I used the same hyperparameters as yours in the part of train_synthia in config.ini. Moreover, I added the fb_weight( 0.2 ）and occ_weight( 12.4 ) to the train process. Meanwhile, I set the mask_occlusion to fb, set the development to True in config.ini. And the rest of the config.ini is the same as yours. The loss curves from my own trained model are shown below.
I think these loss curves aren't very reasonable, such as loss_combined seems not to be decreased efficiently， smooth_2nd_loss may be wrong, and so on. And I'm so confused about the real reason causing this problem. I think the only difference between my own train process and yours is that I use a smaller synthia dataset( I set the development to True in config.ini), but I'm not sure if this difference caused the problem. By the way, I use 4 gpu to train my model, and the loss curves shown above are only the output of tower 0. I hope that you can give me some advices to figure the problem out, and can you show some loss curves obtained from your trained model on synthia dataset? If you can give some loss curves as reference, I think it will be very helpful for me. Thinks so much in advance, best wishes!