search

PingoLH / FCHarDNet

Fully Convolutional HarDNet for Segmentation in Pytorch
MIT License
195 stars 52 forks source link

Error training using Pascal VOC #43

Open ryanaleksander opened 4 years ago

ryanaleksander commented 4 years ago

I've been trying to train the model Pascal VOC 2012 and SBD and these things keep popping up whenever the training process reaches the validation loop (the training part is perfectly fine).

C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [127,0,0], thread: [868,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [124,0,0], thread: [160,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [141,0,0], thread: [166,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [122,0,0], thread: [3,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [137,0,0], thread: [67,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [135,0,0], thread: [167,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [135,0,0], thread: [168,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [141,0,0], thread: [150,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [133,0,0], thread: [167,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [133,0,0], thread: [168,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [137,0,0], thread: [717,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [128,0,0], thread: [868,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [127,0,0], thread: [798,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [139,0,0], thread: [716,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [122,0,0], thread: [154,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [135,0,0], thread: [727,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [137,0,0], thread: [761,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [139,0,0], thread: [745,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [139,0,0], thread: [753,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [107,0,0], thread: [16,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [107,0,0], thread: [17,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [143,0,0], thread: [164,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [133,0,0], thread: [728,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [113,0,0], thread: [42,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [96,0,0], thread: [3,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [96,0,0], thread: [4,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [96,0,0], thread: [6,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [109,0,0], thread: [44,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [109,0,0], thread: [51,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [109,0,0], thread: [947,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [109,0,0], thread: [948,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [118,0,0], thread: [937,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [131,0,0], thread: [796,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [131,0,0], thread: [732,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [100,0,0], thread: [12,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [134,0,0], thread: [167,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [134,0,0], thread: [168,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [138,0,0], thread: [717,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [128,0,0], thread: [798,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [99,0,0], thread: [1023,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [118,0,0], thread: [32,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [118,0,0], thread: [33,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [118,0,0], thread: [57,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [121,0,0], thread: [154,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [100,0,0], thread: [1023,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [125,0,0], thread: [160,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [103,0,0], thread: [14,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [130,0,0], thread: [796,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [130,0,0], thread: [732,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [140,0,0], thread: [150,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [132,0,0], thread: [731,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [138,0,0], thread: [126,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [97,0,0], thread: [3,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [97,0,0], thread: [4,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [97,0,0], thread: [6,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [121,0,0], thread: [36,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [121,0,0], thread: [37,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [121,0,0], thread: [59,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [140,0,0], thread: [745,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [140,0,0], thread: [753,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [115,0,0], thread: [939,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [115,0,0], thread: [940,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [123,0,0], thread: [3,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [105,0,0], thread: [46,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [105,0,0], thread: [47,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [112,0,0], thread: [943,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [112,0,0], thread: [944,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [16,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [17,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [112,0,0], thread: [42,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [112,0,0], thread: [53,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [46,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [47,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [115,0,0], thread: [55,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [103,0,0], thread: [1021,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [144,0,0], thread: [163,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [951,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [106,0,0], thread: [952,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [136,0,0], thread: [761,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [125,0,0], thread: [63,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: block: [136,0,0], thread: [125,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.

The config is mostly just default, I didn't change much aside from the image size. I also supplied the "sdb_path" to the dataloaders.

EDIT: It seems to be caused by the lack of augmentation in valloader. I tried again with augementation and it works fine for some reasons