search

endernewton / tf-faster-rcnn

Tensorflow Faster RCNN for Object Detection
https://arxiv.org/pdf/1702.02138.pdf
MIT License
3.65k stars 1.58k forks source link

InvalidArgumentError: Incompatible shapes: [128,12] vs. [0,12] #429

Open GiserZhy opened 5 years ago

GiserZhy commented 5 years ago 
I'm trying to train my own dataset based on VGG16 network. However, both my datasets & VOC_2007 datasets occur such error while training:

InvalidArgumentError (see above for traceback): Incompatible shapes: [128,12] vs. [0,12] [[Node: gradients/LOSS_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/LOSS_default/Sum_1_grad/Tile, vgg_16_1/rpn_rois/proposal_target/_335)]] [[Node: LOSS_default/Mean_2/_353 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1360_LOSS_default/Mean_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

This error will occur randomly while training more than 10 times. I really don't know why a tensor with shape of [0, 4 * class_num] has appeared. Does anyone know the solution to this problem? Thank you very much.
pandaMingx commented 5 years ago

hello ,I have the same problem.Has your problem been solved?

SimpleXP commented 5 years ago

Hi, same error here.

Lucky for me, I kind of find a method to solve it. (Jump to the end to check the solution)

First, based on my analysis, the error here is caused by both fg_inds and bg_inds are empty. In code file "/lib/layer_utils/proposal_target_layer.py", line 119. The code check the size of fg_inds and bg_inds. If both are empty, it will go to pdb and the training will stop and ask you to debug it. After I comment it out (because I assume this will never happen), I got this "incompatible shapes" error. So I guess the error is because of the size of fg_inds and bg_inds.

Then I checked the code which assign the value to fg_inds and bg_inds, found out that they are controlled by parameters: cfg.TRAIN.FG_THRESH, cfg.TRAIN.BG_THRESH_HI and cfg.TRAIN.BG_THRESH_LO.

In "/lib/model/config.py", those parameters are set to 0.5, 0.5 and 0.1, respectively. The problem is with cfg.TRAIN.BG_THRESH_LO which is 0.1 by default. This means that if the overlap of proposal and the ground truth is larger than 0.1, and less than 0.5, it is considered as background. This will bring a problem, if your ground truth is extremely small, and the overlap area of all the proposals and background is less than 0.1, no proposal will consider as background, and of course, no proposals are consider as foreground either. This will cause fg_inds and bg_inds both empty.

So, to sovle this, I change the parameter cfg.TRAIN.BG_THRESH_LO (line 83 in /lib/model/config.py) from 0.1 to 0.0. Means even if there is no overlap of proposals and groundtruth, it is still consider as background. This will make sure that bg_inds will not be empty.

Hope this will solve you problem as well.

buaaYYC commented 5 years ago

I solve the problem .You need to modify the code demo.py -- net.create_architecture("TEST", len(CLASSES), tag='default', anchor_scales=[8, 16, 32]), i change 21 to len(CLASSES),the demo.py can work.