search

XiaXuehai / faceboxes

faceboxes implement by pytorch
82 stars 22 forks source link

RuntimeError: device-side assert triggered 请教大佬 #19

Closed ZLeopard closed 5 years ago

ZLeopard commented 5 years ago

/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [33,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [37,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [41,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [46,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [50,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [314,0,0], thread: [60,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [315,0,0], thread: [200,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [315,0,0], thread: [219,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [262,0,0], thread: [117,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed. Traceback (most recent call last): File "/home/t1070/TeddyZhang/DEEP_LEARNING/faceboxes-master/trainvisdom.py", line 116, in train() File "/home/t1070/TeddyZhang/DEEP_LEARNING/faceboxes-master/trainvisdom.py", line 73, in train loss = criterion(loc_preds,loc_targets,conf_preds,conf_targets) File "/home/t1070/anaconda2/envs/TeddyZhang/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, *kwargs) File "/home/t1070/TeddyZhang/DEEP_LEARNING/faceboxes-master/multibox_loss.py", line 64, in forward neg = self.hard_negative_mining(conf_loss, pos) # (1621824, (16,21824)) File "/home/t1070/TeddyZhang/DEEP_LEARNING/faceboxes-master/multibox_loss.py", line 29, in hard_negative_mining conf_loss[pos.view(-1,1)] = 0 #去掉正样本,the rest are neg conf_loss RuntimeError: device-side assert triggered

ZLeopard commented 5 years ago

不知道这是为什么？？

XiaXuehai commented 5 years ago

indexValue >= 0 && indexValue < src.sizes[dim],看起来是pos.view(-1,1)这个index条件不满足，设个断点调试下看

ZLeopard commented 5 years ago

我试过了，是满足的，两个tensor维度一致的

ZLeopard commented 5 years ago

@XiaXuehai 你在训练时遇到过着这个问题么？我不知道是电脑显存不够还是什么问题？？

XiaXuehai commented 5 years ago

我没有遇到过。。多debug吧。看看pos.view(-1,1)的值有没有负数之类的吧

ZLeopard commented 5 years ago

@XiaXuehai 好像是有负数，我应该怎么解决呢？我用的是我自己的数据集。。。

ZLeopard commented 5 years ago

@XiaXuehai 而且代码中有个检查inf_loss的函数，我训练时老是回调到里面去，我就把它屏蔽了

XiaXuehai commented 5 years ago

有负数可能是数据的label出了问题，可能是宽度为0之类的。

ZLeopard commented 5 years ago

@XiaXuehai 非常感谢您，您的一句话点醒了我，数据集读取有零值，我剔除了就可以了，万分感谢

yokings commented 4 years ago

@XiaXuehai 好像是有负数，我应该怎么解决呢？我用的是我自己的数据集。。。

pos怎么会有负数，不是True或者False吗