NVIDIA / DIGITS

Deep Learning GPU Training System
https://developer.nvidia.com/digits
BSD 3-Clause "New" or "Revised" License
4.12k stars 1.38k forks source link

semantic-segmentation example error - num of labels don't match predictions #1023

Closed lasandcris closed 8 years ago

lasandcris commented 8 years ago

I followed the walkthrough of the segmentation example - creating the DB and setting up the new model for the project was very straightforward. However, the training throws an error:

Train Caffe Model: Check failed: outernum * innernum == bottom[1]->count() (166000 vs. 498000) Number of labels must match number of predictions; e.g., i* Check failure stack trace: *

Has anyone had this error yet? thanks

gheinrich commented 8 years ago

Hello, are you using single-channel labels? Can you post the label shape in your training and validation databases (this information is available on your dataset job page).

gheinrich commented 8 years ago

Can you also post the full caffe_output.log (from the model page). Thanks.

lasandcris commented 8 years ago

@gheinrich - hi, the log from the model page is:

upscore <- score_fr upscore -> upscore Setting up upscore Top shape: 1 21 415 543 (4732245) Memory required for data: 100907428 Creating layer score Creating Layer score score <- upscore score <- data_data_0_split_1 score -> score Setting up score Top shape: 1 21 375 500 (3937500) Memory required for data: 116657428 Creating layer loss Creating Layer loss loss <- score loss <- label loss -> loss Creating layer loss Check failed: outernum * innernum == bottom[1]->count() (187500 vs. 562500) Number of labels must match number of predictions; e.g.,* Check failure stack trace: *

and my shapes are:

train_db DB Feature shape (3L, 375L, 500L) Label shape (3L, 375L, 500L)

val_db DB Feature shape (3L, 281L, 500L) Label shape (3L, 281L, 500L)

I used the Kitti dataset - using the class folder for labels. Thanks for your help.

gheinrich commented 8 years ago

Did you accidentally create an image processing dataset, by any chance? This model requires that you create an image segmentation dataset.

lasandcris commented 8 years ago

@gheinrich - Thanks Greg!

Was it the label shape that gave it away? Is there a reshape in the segmentation DB creation?

gheinrich commented 8 years ago

Yes, the label shape gave you away :-) labels should be single channel (shape=1*Height*Width). In the general image processing case, paletted images are automatically converted to RGB. In the segmentation case we don't do this (instead we keep separate track of the color palette).

lasandcris commented 8 years ago

@gheinrich - I don't know why I didn't think of that - most segmentation algos encourage single channel maps - maybe some superpixel algos use 3 channels - but thats just completely off topic.

thanks again, laszlo