Open promisechen666 opened 5 months ago
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [0,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [1,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [2,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [3,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [4,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [5,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [6,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [7,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [8,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [9,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [10,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [11,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [12,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [13,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [14,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [15,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [16,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [17,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [18,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [19,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [20,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [21,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [22,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [23,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [24,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [25,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [26,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [27,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [28,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [29,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [30,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [31,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [0,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [1,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [2,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [3,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [4,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [5,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [6,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [7,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [8,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [9,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [10,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [11,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [12,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [13,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [14,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [15,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [16,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [17,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [18,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [19,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [20,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [21,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [22,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [23,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [24,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [25,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [26,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [27,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [28,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [29,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [30,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [31,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. Traceback (most recent call last): File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 157, in train(args, YAMLParser(args.config)) File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 104, in train loss = loss_function(x["flow"], inputs["inp_list"].to(device), inputs["inp_pol_mask"].to(device)) File "/home/yons/anaconda3/envs/DCEIFlow/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/media/yons/F/code/ssl_e2vid-main/loss/flow.py", line 65, in forward fw_iwe_neg = interpolate(fw_idx.long(), fw_weights, self.res, polarity_mask=pol_mask[:, :, 1:2]) File "/media/yons/F/code/ssl_e2vid-main/utils/iwe.py", line 87, in interpolate iwe = torch.zeros((idx.shape[0], res[0] res[1], 1)).to(idx.device) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
When using CPU instead of GPU, the error doesn't occur.
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [0,0,0] Assertion
train(args, YAMLParser(args.config))
File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 104, in train
loss = loss_function(x["flow"], inputs["inp_list"].to(device), inputs["inp_pol_mask"].to(device))
File "/home/yons/anaconda3/envs/DCEIFlow/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, *kwargs)
File "/media/yons/F/code/ssl_e2vid-main/loss/flow.py", line 65, in forward
fw_iwe_neg = interpolate(fw_idx.long(), fw_weights, self.res, polarity_mask=pol_mask[:, :, 1:2])
File "/media/yons/F/code/ssl_e2vid-main/utils/iwe.py", line 87, in interpolate
iwe = torch.zeros((idx.shape[0], res[0] res[1], 1)).to(idx.device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
idx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [1,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [2,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [3,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [4,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [5,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [6,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [7,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [8,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [9,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [10,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [11,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [12,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [13,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [14,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [15,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [16,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [17,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [18,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [19,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [20,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [21,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [22,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [23,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [24,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [25,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [26,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [27,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [28,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [29,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [30,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [35,0,0], thread: [31,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [0,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [1,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [2,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [3,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [4,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [5,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [6,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [7,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [8,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [9,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [10,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [11,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [12,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [13,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [14,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [15,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [16,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [17,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [18,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [19,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [20,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [21,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [22,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [23,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [24,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [25,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [26,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [27,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [28,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [29,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [30,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [39,0,0], thread: [31,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. Traceback (most recent call last): File "/media/yons/F/code/ssl_e2vid-main/train_flow.py", line 157, inWhen using CPU instead of GPU, the error doesn't occur.