search

Julie-tang00 / Point-BERT

[CVPR 2022] Pre-Training 3D Point Cloud Transformers with Masked Point Modeling
MIT License
541 stars 65 forks source link

Pre-training BERT errror #70

Closed JayKarhade closed 1 year ago

JayKarhade commented 1 year ago

Hi, I was trying to run the pre-training command according to the instructions, however i face this error. Any help would be appreciated!


(pmae) jay@jay-System-Product-Name:~/Point-BERT$ bash scripts/train.sh 0    --config cfgs/ShapeNet55_models/dvae.yaml     --exp_name test
+ GPUS=0
+ PY_ARGS='--config cfgs/ShapeNet55_models/dvae.yaml --exp_name test'
+ CUDA_VISIBLE_DEVICES=0
+ python main.py --config cfgs/ShapeNet55_models/dvae.yaml --exp_name test
2023-10-08 22:33:05,627 - dvae - INFO - Copy the Config file from cfgs/ShapeNet55_models/dvae.yaml to ./experiments/dvae/ShapeNet55_models/test/config.yaml
2023-10-08 22:33:05,627 - dvae - INFO - args.config : cfgs/ShapeNet55_models/dvae.yaml
2023-10-08 22:33:05,627 - dvae - INFO - args.launcher : none
2023-10-08 22:33:05,627 - dvae - INFO - args.local_rank : 0
2023-10-08 22:33:05,627 - dvae - INFO - args.num_workers : 4
2023-10-08 22:33:05,627 - dvae - INFO - args.seed : 0
2023-10-08 22:33:05,627 - dvae - INFO - args.deterministic : False
2023-10-08 22:33:05,627 - dvae - INFO - args.sync_bn : False
2023-10-08 22:33:05,627 - dvae - INFO - args.exp_name : test
2023-10-08 22:33:05,627 - dvae - INFO - args.start_ckpts : None
2023-10-08 22:33:05,627 - dvae - INFO - args.ckpts : None
2023-10-08 22:33:05,627 - dvae - INFO - args.val_freq : 1
2023-10-08 22:33:05,627 - dvae - INFO - args.resume : False
2023-10-08 22:33:05,627 - dvae - INFO - args.test : False
2023-10-08 22:33:05,627 - dvae - INFO - args.finetune_model : False
2023-10-08 22:33:05,627 - dvae - INFO - args.scratch_model : False
2023-10-08 22:33:05,627 - dvae - INFO - args.label_smoothing : False
2023-10-08 22:33:05,627 - dvae - INFO - args.mode : None
2023-10-08 22:33:05,627 - dvae - INFO - args.way : -1
2023-10-08 22:33:05,627 - dvae - INFO - args.shot : -1
2023-10-08 22:33:05,627 - dvae - INFO - args.fold : -1
2023-10-08 22:33:05,627 - dvae - INFO - args.experiment_path : ./experiments/dvae/ShapeNet55_models/test
2023-10-08 22:33:05,627 - dvae - INFO - args.tfboard_path : ./experiments/dvae/ShapeNet55_models/TFBoard/test
2023-10-08 22:33:05,627 - dvae - INFO - args.log_name : dvae
2023-10-08 22:33:05,627 - dvae - INFO - args.use_gpu : True
2023-10-08 22:33:05,627 - dvae - INFO - args.distributed : False
2023-10-08 22:33:05,627 - dvae - INFO - config.optimizer = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.optimizer.type : AdamW
2023-10-08 22:33:05,627 - dvae - INFO - config.optimizer.kwargs = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.optimizer.kwargs.lr : 0.0005
2023-10-08 22:33:05,627 - dvae - INFO - config.optimizer.kwargs.weight_decay : 0.0005
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler.type : CosLR
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler.kwargs = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler.kwargs.epochs : 300
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler.kwargs.initial_epochs : 10
2023-10-08 22:33:05,627 - dvae - INFO - config.scheduler.kwargs.warming_up_init_lr : 5e-05
2023-10-08 22:33:05,627 - dvae - INFO - config.temp = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.temp.start : 1
2023-10-08 22:33:05,627 - dvae - INFO - config.temp.target : 0.0625
2023-10-08 22:33:05,627 - dvae - INFO - config.temp.ntime : 100000
2023-10-08 22:33:05,627 - dvae - INFO - config.kldweight = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.kldweight.start : 0
2023-10-08 22:33:05,627 - dvae - INFO - config.kldweight.target : 0.1
2023-10-08 22:33:05,627 - dvae - INFO - config.kldweight.ntime : 100000
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train._base_ = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train._base_.NAME : ShapeNet
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train._base_.DATA_PATH : data/ShapeNet55-34/ShapeNet-55
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train._base_.N_POINTS : 8192
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train._base_.PC_PATH : data/ShapeNet55-34/shapenet_pc
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train.others = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train.others.subset : train
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train.others.npoints : 1024
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.train.others.bs : 64
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val._base_ = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val._base_.NAME : ShapeNet
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val._base_.DATA_PATH : data/ShapeNet55-34/ShapeNet-55
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val._base_.N_POINTS : 8192
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val._base_.PC_PATH : data/ShapeNet55-34/shapenet_pc
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val.others = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val.others.subset : test
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val.others.npoints : 1024
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.val.others.bs : 1
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test._base_ = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test._base_.NAME : ShapeNet
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test._base_.DATA_PATH : data/ShapeNet55-34/ShapeNet-55
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test._base_.N_POINTS : 8192
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test._base_.PC_PATH : data/ShapeNet55-34/shapenet_pc
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test.others = edict()
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test.others.subset : test
2023-10-08 22:33:05,627 - dvae - INFO - config.dataset.test.others.npoints : 1024
2023-10-08 22:33:05,628 - dvae - INFO - config.dataset.test.others.bs : 1
2023-10-08 22:33:05,628 - dvae - INFO - config.model = edict()
2023-10-08 22:33:05,628 - dvae - INFO - config.model.NAME : DiscreteVAE
2023-10-08 22:33:05,628 - dvae - INFO - config.model.group_size : 32
2023-10-08 22:33:05,628 - dvae - INFO - config.model.num_group : 64
2023-10-08 22:33:05,628 - dvae - INFO - config.model.encoder_dims : 256
2023-10-08 22:33:05,628 - dvae - INFO - config.model.num_tokens : 8192
2023-10-08 22:33:05,628 - dvae - INFO - config.model.tokens_dims : 256
2023-10-08 22:33:05,628 - dvae - INFO - config.model.decoder_dims : 256
2023-10-08 22:33:05,628 - dvae - INFO - config.total_bs : 64
2023-10-08 22:33:05,628 - dvae - INFO - config.step_per_update : 1
2023-10-08 22:33:05,628 - dvae - INFO - config.max_epoch : 300
2023-10-08 22:33:05,628 - dvae - INFO - config.consider_metric : CDL1
2023-10-08 22:33:05,628 - dvae - INFO - Distributed training: False
2023-10-08 22:33:05,628 - dvae - INFO - Set random seed to 0, deterministic: False
2023-10-08 22:33:05,628 - ShapeNet-55 - INFO - [DATASET] sample out 1024 points
2023-10-08 22:33:05,628 - ShapeNet-55 - INFO - [DATASET] Open file data/ShapeNet55-34/ShapeNet-55/train.txt
2023-10-08 22:33:05,647 - ShapeNet-55 - INFO - [DATASET] 41952 instances were loaded
2023-10-08 22:33:05,647 - ShapeNet-55 - INFO - [DATASET] sample out 1024 points
2023-10-08 22:33:05,647 - ShapeNet-55 - INFO - [DATASET] Open file data/ShapeNet55-34/ShapeNet-55/test.txt
2023-10-08 22:33:05,653 - ShapeNet-55 - INFO - [DATASET] 10518 instances were loaded
2023-10-08 22:33:07,624 - dvae - INFO - Using Data parallel ...
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [0,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [46,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [48,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [51,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [52,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [53,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [54,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [55,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [145,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [36,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [46,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [48,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [51,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [52,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [53,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [54,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [55,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [154,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [126,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "main.py", line 72, in <module>
    main()
  File "main.py", line 68, in main
    run_net(args, config, train_writer, val_writer)
  File "/home/jay/Point-BERT/tools/runner.py", line 130, in run_net
    ret = base_model(points, temperature = temp, hard = False)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jay/Point-BERT/models/dvae.py", line 335, in forward
    logits = self.dgcnn_1(logits, center) #  B G N
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jay/Point-BERT/models/dvae.py", line 87, in forward
    f = self.layer1(f)                           # B 256 N k
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/jay/miniconda3/envs/pmae/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 442, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([64, 256, 64, 4], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(256, 256, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0xd9204e20
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 64, 256, 64, 4, 
    strideA = 65536, 256, 4, 1, 
output: TensorDescriptor 0xd9204760
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 64, 256, 64, 4, 
    strideA = 65536, 256, 4, 1, 
weight: FilterDescriptor 0xd92039a0
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 256, 256, 1, 1, 
Pointer addresses: 
    input: 0x7f2f6d200000
    output: 0x7f2f6c000000
    weight: 0x7f311ca00000
JayKarhade commented 1 year ago

Update : It seems that the Knn cuda library does not work properly on my pc and was giving the error.

Possible solutions are to switch to pytorch3d's KNN functions or use faiss/torch knn functions and loop over each batch.