AttributeError: 'NoneType' object has no attribute 'maxsize'

[wakakaFF]

initial lr is 0.01 scheduler_type is poly, warmup steps is 0 Epoch 0: 0%| | 0/250 [00:00<?, ?it/s]using pin_memory on device 0 ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [423,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [680,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [681,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [682,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [551,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [552,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [553,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [554,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [810,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [811,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [812,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [813,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [169,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [171,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [172,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [296,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [301,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [302,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [682,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [683,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [684,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [685,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [947,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [933,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [934,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [935,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [936,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [937,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [938,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [939,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [940,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [172,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [173,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [174,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [175,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [176,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [812,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [813,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [814,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [815,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [816,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [817,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [818,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [49,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [50,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [51,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [52,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [425,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [426,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [427,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [428,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [429,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [430,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [431,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [432,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [552,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [553,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [554,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [555,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [556,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [557,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [558,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [559,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [560,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [562,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [38,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [39,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [40,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [41,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [42,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [43,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [44,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [49,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [43,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [44,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [45,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [46,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [47,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [48,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [816,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [817,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [818,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [819,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [820,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [821,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [822,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [304,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [305,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [306,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [309,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [310,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [805,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [806,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [808,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [809,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [810,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [935,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [936,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [937,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [938,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [941,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [942,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [557,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [558,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [559,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [560,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [561,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [562,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [563,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [940,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [941,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [942,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [943,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [944,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [799,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [671,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [684,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [685,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [686,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [687,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [688,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [689,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [690,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [691,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [432,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [433,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [434,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [435,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [175,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [176,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [177,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [178,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [179,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [180,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [181,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [182,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [688,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [689,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [690,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [691,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [692,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [576,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [960,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [159,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [704,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [287,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [927,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [320,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [192,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [415,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [543,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [448,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [159,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [64,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [173,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [30,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [431,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [432,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [434,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [927,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [415,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [38,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [39,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [41,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [42,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [44,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [45,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [46,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [671,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [832,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [414,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [415,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [543,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [543,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [799,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [287,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [30,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [157,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [158,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [1,0,0], thread: [159,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [300,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [301,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [427,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [428,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [287,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [64,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [556,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [687,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [671,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [811,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [812,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [813,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [814,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [820,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [704,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [799,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [192,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [927,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [320,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [64,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [181,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [448,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [311,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [312,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [315,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [316,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [3,0,0], thread: [448,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [936,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [937,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [948,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [36,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [47,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [174,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [557,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [427,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [428,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [429,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [52,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [53,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [55,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [158,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [159,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [414,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [286,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [287,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [671,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [960,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [299,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [5,0,0], thread: [300,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [4,0,0], thread: [543,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [576,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [832,0,0] Assertion t >= 0 && t < n_classes failed. ../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [2,0,0], thread: [192,0,0] Assertion t >= 0 && t < n_classes failed. Epoch 0: 0%| | 0/250 [00:06<?, ?it/s] Traceback (most recent call last): File "/mnt/workspace/models/SegMamba/3_train.py", line 174, in trainer.train(train_dataset=train_ds, val_dataset=val_ds) File "/mnt/workspace/models/SegMamba/light_training/trainer.py", line 415, in train self.train_epoch( File "/mnt/workspace/models/SegMamba/light_training/trainer.py", line 458, in train_epoch loss = self.training_step(batch) File "/mnt/workspace/models/SegMamba/3_train.py", line 69, in training_step self.log("training_loss", loss, step=self.global_step) File "/mnt/workspace/models/SegMamba/light_training/trainer.py", line 506, in log self.writer.add_scalar(k, scalar_value=v, global_step=step) File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/writer.py", line 393, in add_scalar summary = scalar( File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/summary.py", line 369, in scalar tensor = make_np(tensor).squeeze() File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/_convert_np.py", line 23, in make_np return _prepare_pytorch(x) File "/usr/local/lib/python3.10/dist-packages/torch/utils/tensorboard/_convert_np.py", line 30, in _prepare_pytorch x = x.detach().cpu().numpy() RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception in thread Thread-12 (results_loop): Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.10/dist-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop raise e File "/usr/local/lib/python3.10/dist-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the " RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message Exception ignored in: <function NonDetMultiThreadedAugmenter.del at 0x7f1da02bea70> Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 253, in del File "/usr/local/lib/python3.10/dist-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 238, in _finish File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 338, in set File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 297, in notify_all AttributeError: 'NoneType' object has no attribute 'maxsize'

[wakakaFF]

你好，前辈。我完全按照您的步骤且使用相同的数据集，在跑了5个batchsize后出现这样的报错，我现在有点怀疑是不是我传数据集到aliyun的时候出现了丢失。

[ge-xing]

你的输出都是不对 (1, 1, 128, 128, 128)是输出吗？ 应该是(1, 4, 128, 128, 128)

[wakakaFF]

我这里打印的是label的维度

[ge-xing]

你打印下np.unique(labels.cpu().numpy()) ，看看报错的时候labels都包含哪几个数字

[wakakaFF]

坏了，还真是出现了4.....

[wakakaFF]

可是为什么会出现4呢，我数据集（barts2023）和步骤都和readme里的一摸一样啊。

[ge-xing]

你的数据集不对，brats23里面不可能有4，你估计是下载错数据集了 重新下载一下吧

[ge-xing]

可能混进去了其他数据，比如brats2020

[wakakaFF]

好的，我再去看看，谢谢你。

[NetfriendT]

@wakakaFF 你好，请问我在更换了数据集之后2.预处理一直报错说找不到文件是怎么回事呢？而且我是按照原Bra数据集的位置放置的文件修改的代码。 他最终读取的文件The file "./data/raw data/training/patient084/4d.nii.gz" does not exist.也不对，应该是patient001_4d.nii.gz才行

[wakakaFF]

@wakakaFF 你好，请问我在更换了数据集之后2.预处理一直报错说找不到文件是怎么回事呢？而且我是按照原Bra数据集的位置放置的文件修改的代码。 他最终读取的文件The file "./data/raw data/training/patient084/4d.nii.gz" does not exist.也不对，应该是patient001_4d.nii.gz才行

问题应该出现在重命名的部分，也就是“1_rename_mri_data.py”里，因为数据集的命名规则不一样，你看一下应该能看懂。

[NetfriendT]

@wakakaFF 是的，已经发现了这个问题，感谢你的帮助

[NetfriendT]

您好，最近在训练时出现了一个现象，训练到一半老是会报错内存问题，但是我使用的服务器账号并没有跑太多任务，此外还会报错数据维度的问题。我只输入了一个图像和一个标签，我认为我把segmamba输入改成1输出为4不变应该是正确的。奇怪的地方在于有的时候直接报错说维度问题并且报错内存问题，有的时候二选一报错，有的时候训练都进行了40%了又报错，我感觉数据都开始训练了为什么还会说数据维度不对呢？那怎么进去模型训练呢？请问您有遇到过这种问题吗 并且就算我把segmamba改为in_chans=4,out_chans=4,依然会报错 could not broadcast input array from shape (4,128,128,128) into shape (1,128,128,128)

[wakakaFF]

您好，最近在训练时出现了一个现象，训练到一半老是会报错内存问题，但是我使用的服务器账号并没有跑太多任务，此外还会报错数据维度的问题。我只输入了一个图像和一个标签，我认为我把segmamba输入改成1输出为4不变应该是正确的。奇怪的地方在于有的时候直接报错说维度问题并且报错内存问题，有的时候二选一报错，有的时候训练都进行了40%了又报错，我感觉数据都开始训练了为什么还会说数据维度不对呢？那怎么进去模型训练呢？请问您有遇到过这种问题吗 并且就算我把segmamba改为in_chans=4,out_chans=4,依然会报错 could not broadcast input array from shape (4,128,128,128) into shape (1,128,128,128)

我还没有遇到过这类问题。

[NetfriendT]

好的，感谢您的回答