search

alexgkendall / SegNet-Tutorial

Files for a tutorial to train SegNet for road scenes using the CamVid dataset
http://mi.eng.cam.ac.uk/projects/segnet/tutorial.html
847 stars 517 forks source link

Error in Training #22

Closed chuhang closed 8 years ago

chuhang commented 8 years ago

Hi,

I am trying to train a segmentation network with 5 categories. I have prepared my images and labels and modified the path.

The only change I made to the segnet_train.prototext file is: num_output and loss_param.

However, when I start training I got either 'caffe free(): invalide next size' or 'CUBLAS_INTERNAL_ERROR'.

I can run the training command listed on the tutorial without error.

What would be the problem here? Thanks in advance!

alexgkendall commented 8 years ago

In my experience the CUBLAS error is when you have training data that exceeds your output classes. If you are outputting 5 classes from your final convolutional layer, then your label data should be in the range 0 - 4 (inclusive).

Often running in CPU mode may give you a more descriptive error too?

Alex

ghost commented 8 years ago

Hello, @alexgkendall. I have the same issue above. I labelled data from 0 - 4 (inclusive). But I cropped my data, and some of data (because of cropping) contain only some classes (for instance only 0 and 1). Could this be a source of the problem?

alexgkendall commented 8 years ago

No, this is fine to have only some of the classes present, this is the case in CamVid too.

Could you please paste your model file?

ghost commented 8 years ago

Hello, @alexgkendall. Thank you for your quick responses! Actually ... maybe the issue was with Caffe configuration. In our server, we re-installed the caffe-segnet, and re-ran the experiment ... and it worked!!!

Thank you again!