Bobholamovic
commented
5 months ago 你好，建议检查一下类别数是否设置正确。例如，数据集标签的label是否可能大于等于总类别数。
Open julinfn opened 12 months ago
Bobholamovic
commented
5 months ago 你好，建议检查一下类别数是否设置正确。例如，数据集标签的label是否可能大于等于总类别数。
Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertion
model.train(
File "e:\git\paddle\PaddleRS\need\paddlers\tasks\object_detector.py", line 350, in train
return self._real_train(args)
File "e:\git\paddle\PaddleRS\need\paddlers\tasks\object_detector.py", line 454, in _real_train
self.train_loop(
File "e:\git\paddle\PaddleRS\need\paddlers\tasks\base.py", line 402, in train_loop
outputs = self.train_step(step, data, net, optimizer)
File "e:\git\paddle\PaddleRS\need\paddlers\tasks\base.py", line 788, in train_step
outputs = self.run(net, data, mode='train')
File "e:\git\paddle\PaddleRS\need\paddlers\tasks\object_detector.py", line 157, in run
net_out = net(inputs)
File "D:\profProgram\python3912\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call
return self._dygraph_call_func(*inputs, *kwargs)
File "D:\profProgram\python3912\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func
outputs = self.forward(inputs, kwargs)
File "e:\git\paddle\PaddleRS\need\paddlers\models\ppdet\modeling\architectures\meta_arch.py", line 60, in forward
out = self.get_loss()
File "e:\git\paddle\PaddleRS\need\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 112, in get_loss
rpn_loss, bbox_loss = self._forward()
File "e:\git\paddle\PaddleRS\need\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 80, in _forward
rois, rois_num, rpn_loss = self.rpn_head(body_feats, self.inputs)
File "D:\profProgram\python3912\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call
return self._dygraph_call_func(*inputs, *kwargs)
File "D:\profProgram\python3912\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func
outputs = self.forward(inputs, **kwargs)
File "e:\git\paddle\PaddleRS\need\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py", line 143, in forward
loss = self.get_loss(scores, deltas, anchors, inputs)
File "e:\git\paddle\PaddleRS\need\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py", line 281, in get_loss
pos_ind = paddle.nonzero(pos_mask)
File "D:\profProgram\python3912\lib\site-packages\paddle\tensor\search.py", line 402, in nonzero
outs = _C_ops.where_index(x)
OSError: (External) CUDA error(719), unspecified launch failure.
[Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be syst system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA proc, the process must be terminated and relaunched.] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:251)
[operator < where_index > error]
scatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1044417757] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043999828] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1038362748] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1028488216] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1041107507] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1038512690] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1025719731] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1062530930] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1046412433] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1040383832] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1037096216] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1052784226] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1050774087] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043101528] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1038468011] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1042905369] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1041904647] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1056778009] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1034192824] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1046018282] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1037947415] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1029079965] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1030685376] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1039469801] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1046026074] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026127552] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1038185411] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1039719342] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1046856767] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1031351458] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1037857740] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1052852273] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043838258] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1027262336] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1041238806] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026994516] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1028472681] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1029024146] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043505363] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1042091129] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1047327143] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1029984309] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1031696397] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026858712] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1042069315] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1042087329] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1041296117] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1053384985] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1039710912] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1067889639] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1028286762] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1039959388] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1040023678] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1057672049] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043497574] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1058031812] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026028437] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1043121965] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1039394453] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026286999] Error: ../paddle/phi/kernels/funcs/scatter.cu.h:66 Assertionscatter_i >= 0failed. The index is out of bounds, please check whether the dimensions of index and input meet the requirements. It should be greater than or equal to 0, but received [-1026556020] Traceback (most recent call last): File "e:\git\paddle\PaddleRS\need\faster_rcnn.py", line 75, in