search

golnazghiasi / LRR

code and models for "Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation"
59 stars 14 forks source link

training demo doesn't work #2

Open cslxiao opened 8 years ago

cslxiao commented 8 years ago

When I run the training demo LRR4xTrainVGG16Pascal, it goes fine at the beginning. However, after several iterations, I got the following output:

obj_dil_seg32x: NaN obj_ero_seg32x: NaN objective_32x: NaN

What's wrong with it? I didn't change anything expect for the directories of the dataset.

golnazghiasi commented 8 years ago

I'm not sure why this is happening. I remember I had the exact same problem with the training code of fcn-matconvet (https://github.com/vlfeat/matconvnet-fcn). I could resolve the problem by using a smaller learning rate for just first few epochs. Maybe you can also try this. Can I ask what version of Matconvnet do you use? Also what is your gpu?

cslxiao commented 8 years ago

Thanks @golnazghiasi . I used the latest version of Matconvnet (master branch) on the github. And I am using Titan X.