Cudnn error when using SPADE

emanueledelsozzo commented 5 years ago

Hello, I am trying to use SPADE on a custom dataset. Here the command I run: python train.py --name myTest --dataset_mode custom --label_dir datasets/label_dir --image_dir datasets/image_dir --no_instance --gpu_ids 0,1 --batchSize 2

However, after the creation of the web directory, I get this error:

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [21,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [23,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [24,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [25,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [26,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [27,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [28,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [29,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [30,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [31,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [90,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]` failed. Traceback (most recent call last): File "train.py", line 40, in trainer.run_generator_one_step(data_i) File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step g_losses, generated = self.pix2pix_model(data, mode='generator') File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply raise output File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker output = module(*input, *kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 46, in forward input_semantics, real_image) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 137, in compute_generator_loss input_semantics, real_image, compute_kld_loss=self.opt.use_vae) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 196, in generate_fake fake_image = self.netG(input_semantics, z=z) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/generator.py", line 89, in forward x = self.fc(x) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, **kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:564) frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f25a3999441 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f25a3998d7a in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #2: + 0x13652 (0x7f25a2f45652 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x50 (0x7f25a3989ce0 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #4: + 0x30facb (0x7f2573acdacb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #5: + 0x1420bb (0x7f25a3f270bb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #6: + 0x3c30f4 (0x7f25a41a80f4 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #7: + 0x3c3141 (0x7f25a41a8141 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #24: __libc_start_main + 0xe7 (0x7f25a8297b97 in /lib/x86_64-linux-gnu/libc.so.6) Aborted (core dumped) ` I am using python 3.6, CUDA 10.0, Torch 1.1.0, CuDNN 7.5.0 I have also tried on another machine with CUDA 9.0, Torch 1.0.0, CuDNN 7.4.1, but I got the same error. Could you help me?

alexsannikoff commented 5 years ago

@emanueledelsozzo Check that in your custom dataset doesn't exist class with label > num_labels in your config. Add CUDA_LAUNCH_BLOCKING=1 before python ... and see normal traceback.

emanueledelsozzo commented 5 years ago

Hello @alexsannikoff and thank you for your reply! Here the log I get adding CUDA_LAUNCH_BLOCKING=1

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [450,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [454,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [457,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [463,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [464,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [465,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [466,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [467,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [468,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [469,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [470,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [471,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [472,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [473,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [474,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [475,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [476,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [477,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [478,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [479,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [416,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [417,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [418,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [419,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [420,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [421,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [422,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [423,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [424,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [425,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [426,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [427,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [428,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [429,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [430,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [431,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [432,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [434,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [436,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [114,0,0], thread: [447,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [416,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [417,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [418,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [419,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [420,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [421,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [422,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [423,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [424,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [425,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [426,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [427,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [428,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [429,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [430,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [431,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [432,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [433,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [434,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [435,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [436,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [437,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [438,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [441,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [442,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [443,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [444,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [445,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [446,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [447,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [256,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [257,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [258,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [259,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [260,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [261,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [262,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [263,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [264,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [265,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [266,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [267,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [268,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [269,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [270,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [271,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [273,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [274,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [275,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [276,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [277,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [263,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [265,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [266,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [267,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [268,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [269,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [270,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [271,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [272,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [273,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [274,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [275,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [276,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [277,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [278,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [279,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [280,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [281,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [282,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [283,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [284,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [285,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [286,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [287,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [387,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [388,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [389,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [399,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [400,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [401,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [402,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [403,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [404,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [405,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [406,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [407,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [408,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [409,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [410,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [411,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [412,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [413,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [414,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [188,0,0], thread: [415,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [450,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [451,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [452,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [453,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [454,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [455,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [456,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [457,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [458,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [459,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [460,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [461,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [462,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [463,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [464,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [465,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [466,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [467,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [468,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [469,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [470,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [471,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [472,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [473,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [474,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [475,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [476,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [477,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [478,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]failed. /pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [108,0,0], thread: [479,0,0] AssertionindexValue >= 0 && indexValue < tensor.sizes[dim]` failed. THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorScatterGather.cu line=380 error=59 : device-side assert triggered Traceback (most recent call last): File "train.py", line 40, in trainer.run_generator_one_step(data_i) File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step g_losses, generated = self.pix2pix_model(data, mode='generator') File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply raise output File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker output = module(*input, *kwargs) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 42, in forward input_semantics, real_image = self.preprocess_input(data) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 123, in preprocess_input input_semantics = inputlabel.scatter(1, label_map, 1.0) RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:380 terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:564) frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fc95a205441 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fc95a204d7a in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #2: + 0x13652 (0x7fc9597b1652 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x50 (0x7fc95a1f5ce0 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libc10.so) frame #4: + 0x30facb (0x7fc94fb48acb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #5: + 0x1420bb (0x7fc95a7930bb in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #6: + 0x3c30f4 (0x7fc95aa140f4 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #7: + 0x3c3141 (0x7fc95aa14141 in /home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #24: __libc_start_main + 0xe7 (0x7fc95eb03b97 in /lib/x86_64-linux-gnu/libc.so.6) Aborted (core dumped)` Probably the problem is related to your observation about the labels. What I am trying to do is to train a pix2pix able to learn the transformation from A to B. I am not really using any semantic information. I trained a similar network with pix2pixhd and it worked. Now I would like to do the same with SPADE, but I am not sure whether it is possible or not. I also tried to set netG to pix2pixhd, but I got the same error. By the way, when I trained the pix2pixhd, I used to set label_nc to 0, as suggested in the pix2pixhd README. Here, if I try to do the same, I get this error: `Traceback (most recent call last): File "train.py", line 24, in trainer = Pix2PixTrainer(opt) File "/home/emanuele/repos/testSPADE/SPADE/trainers/pix2pix_trainer.py", line 19, in __init__ self.pix2pix_model = Pix2PixModel(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 25, in __init__ self.netG, self.netD, self.netE = self.initialize_networks(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 92, in initialize_networks netG = networks.define_G(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/__init__.py", line 52, in define_G return create_network(netG_cls, opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/__init__.py", line 41, in create_network net = cls(opt) File "/home/emanuele/repos/testSPADE/SPADE/models/networks/generator.py", line 38, in __init__ self.fc = nn.Conv2d(self.opt.semantic_nc, 16 * nf, 3, padding=1) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 327, in __init__ False, _pair(0), groups, bias, padding_mode) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 45, in __init__ self.reset_parameters() File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 48, in reset_parameters init.kaiming_uniform_(self.weight, a=math.sqrt(5)) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 323, in kaiming_uniform_ fan = _calculate_correct_fan(tensor, mode) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 292, in _calculate_correct_fan fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor) File "/home/emanuele/anaconda3/envs/SPADEEnv/lib/python3.6/site-packages/torch/nn/init.py", line 222, in _calculate_fan_in_and_fan_out receptive_field_size = tensor[0][0].numel() IndexError: index 0 is out of bounds for dimension 0 with size 0`

aviel08 commented 5 years ago

Hi, I'm trying to train a custom dataset as well and I'm facing the same error. In my case I'm using labels and instances but that doesn't make any difference.

alexsannikoff commented 5 years ago

@emanueledelsozzo File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 42, in forward input_semantics, real_image = self.preprocess_input(data) File "/home/emanuele/repos/testSPADE/SPADE/models/pix2pix_model.py", line 123, in preprocess_input input_semantics = input_label.scatter(1, label_map, 1.0)

Here programm try to create one-hot encoding your semantic mask. And if num_classes on your mask > label_nc - this error occured.

What about learning image translation from A to B. In this code I think in pix2pixhd model using SPADE block, which use semantic mask for input. I think, that in this version you can't learn image translation without change code. The main idea this repo in SPADE block.

alexsannikoff commented 5 years ago

@aviel08 Check your semantic mask that max(np.unique(mask)) < label_nc, and don't forget that instance mask has more classes than semantic and using to separate objects with same class each other. label_nc - number of classes in semantic mask.

emanueledelsozzo commented 5 years ago

@alexsannikoff ok, thank you for your clarification!

vicmancr commented 4 years ago

@emanueledelsozzo Did you get it to work? I am having the same problem. My labels have 3 semantic labels and the background.

I don't get how the labels are treated, since they are loaded in the range [0, 255] and then converted to one-hot encoding. But what happens if some label is missing in a specific image?

vdhyani96 commented 4 years ago

Hi guys, just in case it helps anyone. I was also facing the same issues about CUDA error and was able to resolve it by using --no_instance and --contain_dontcare_label arguments. Also, want to mention that I am not using instance maps with my custom dataset.

zhangdan8962 commented 4 years ago

@vdhyani96 I am wondering if you are doing Image->Image or Semantic->Image task?

vdhyani96 commented 4 years ago

@zhangdan8962 I am doing semantic->image task

Reheiznekrok commented 4 years ago

Hello, I had the same error when attempting to train with a custom data set. I made some changes to my data set and now it works. I don't know if it was just one or all of my changes that made the difference, but for the ones who are as desperate as me, here are all my changes:

Make sure the data set has the exact same structure as in datasets/coco_stuff/, i.e. save images as .jpg, labels as .png, both saved in separate folders. The images should have 3 channels (last), the labels are only of shape width x height with labels as gray scale values. Names should consist of 12 digits, counting from bottom up, e.g. 000000000184.jpg and must be identical for images and labels
use the option --contains_dontcare_label and potentially a very high --labels_nc (and --no_instance if you don't have this in your data set)

General approach: first make it work on datasets/coco_stuff/ with --dataset_mode coco, then try coco_stuff as custom data set and then compare to your own. Hope this helps someone, cheers

liping1997 commented 1 year ago

can be solved by "python train.py --name work --dataset_mode custom --label_dir ./datasets/label --image_dir ./datasets/image --no_instance --contain_dontcare_label --label_nc 255" the --contain_dontcare_label and --label_nc 255 must both be added

Rajeswari-gn commented 1 year ago

I have tried all the above solutions still I am getting the same error. Has anyone tried some other solutions which helped to sort out this issue?

I have no issues using the coco_stuff dataset, but I get this error only when using the custom dataset.

liping1997 commented 1 year ago

谢谢，已收到。

NVlabs / SPADE

Cudnn error when using SPADE #57