search

SamsungLabs / fcaf3d

[ECCV2022] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection
MIT License
231 stars 37 forks source link

The error about training the S3DIS dataset #40

Closed week0425 closed 2 years ago

week0425 commented 2 years ago

Hello, an error ocurred when I tried to train the S3DIS dataset:

Traceback (most recent call last): File "tools/train.py", line 223, in main() File "tools/train.py", line 212, in main train_model( File "/mmdetection3d/mmdet3d/apis/train.py", line 27, in train_model train_detector( File "/opt/conda/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector runner.run(data_loaders, cfg.workflow) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run epoch_runner(data_loaders[i], kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True, kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "/opt/conda/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step return self.module.train_step(inputs[0], kwargs[0]) File "/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 237, in train_step losses = self(data) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(args, kwargs) File "/mmdetection3d/mmdet3d/models/detectors/base.py", line 58, in forward return self.forward_train(kwargs) File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 48, in forward_train x = self.extract_feat(points, img_metas) File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 40, in extract_feat x = self.neck_with_head(x) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 102, in forward x = self._prune(x, scores) File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 124, in _prune prune_mask[permutation[mask]] = True RuntimeError: invalid shape dimension -255

I don't know how to solve it. Looking forward to your reply!

filaPro commented 2 years ago

Hi @week0425 , Can you please attach the complete .log file and the command you are running?

week0425 commented 2 years ago

Command: python tools/train.py configs/fcaf3d/fcaf3d_s3dis-3d-5class.py The error ocurred at the start of the first training session:

root@adc44dc27719:/mmdetection3d# python tools/train.py configs/fcaf3d/fcaf3d_s3dis-3d-5class.py
/opt/conda/lib/python3.8/site-packages/MinkowskiEngine/__init__.py:36: UserWarning: The environment variable `OMP_NUM_THREADS` not set. MinkowskiEngine will automatically set `OMP_NUM_THREADS=16`. If you want to set `OMP_NUM_THREADS` manually, please export it on the command line before running a python script. e.g. `export OMP_NUM_THREADS=12; python your_program.py`. It is recommended to set it below 24.
  warnings.warn(
2022-07-26 06:44:37,904 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.8.8 (default, Feb 24 2021, 21:46:12) [GCC 7.3.0]
CUDA available: True
GPU 0: Quadro P5000
CUDA_HOME: /usr/local/cuda-11.1
NVCC: Build cuda_11.1.TC455_06.29190527_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.8.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.9.0
OpenCV: 4.5.5
MMCV: 1.3.9
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.15.0+69d12eb
------------------------------------------------------------

2022-07-26 06:44:39,058 - mmdet - INFO - Distributed training: False
2022-07-26 06:44:40,266 - mmdet - INFO - Config:
voxel_size = 0.01
model = dict(
    type='SingleStageSparse3DDetector',
    voxel_size=0.01,
    backbone=dict(type='MEResNet3D', in_channels=3, depth=34),
    neck_with_head=dict(
        type='Fcaf3DNeckWithHead',
        in_channels=(64, 128, 256, 512),
        out_channels=128,
        pts_threshold=100000,
        n_classes=5,
        n_reg_outs=6,
        voxel_size=0.01,
        assigner=dict(type='Fcaf3DAssigner', limit=27, topk=18, n_scales=4),
        loss_bbox=dict(type='IoU3DLoss', loss_weight=1.0, with_yaw=False)),
    train_cfg=dict(),
    test_cfg=dict(nms_pre=1000, iou_thr=0.5, score_thr=0.01))
optimizer = dict(type='AdamW', lr=0.001, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=10, norm_type=2))
lr_config = dict(policy='step', warmup=None, step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=50)
custom_hooks = [dict(type='EmptyCacheHook', after_iter=True)]
checkpoint_config = dict(interval=1, max_keep_ckpts=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/fcaf3d_s3dis-3d-5class'
load_from = None
resume_from = None
workflow = [('train', 1)]
n_points = 100000
dataset_type = 'S3DISDataset'
data_root = './data/s3dis/'
class_names = ('table', 'chair', 'sofa', 'bookcase', 'board')
train_area = [1, 2, 3, 4, 6]
test_area = 5
train_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='DEPTH',
        shift_height=False,
        load_dim=6,
        use_dim=[0, 1, 2, 3, 4, 5]),
    dict(type='LoadAnnotations3D'),
    dict(type='IndoorPointSample', num_points=100000),
    dict(
        type='RandomFlip3D',
        sync_2d=False,
        flip_ratio_bev_horizontal=0.5,
        flip_ratio_bev_vertical=0.5),
    dict(
        type='GlobalRotScaleTrans',
        rot_range=[-0.087266, 0.087266],
        scale_ratio_range=[0.9, 1.1],
        translation_std=[0.1, 0.1, 0.1],
        shift_height=False),
    dict(
        type='DefaultFormatBundle3D',
        class_names=('table', 'chair', 'sofa', 'bookcase', 'board')),
    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
test_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='DEPTH',
        shift_height=False,
        load_dim=6,
        use_dim=[0, 1, 2, 3, 4, 5]),
    dict(
        type='MultiScaleFlipAug3D',
        img_scale=(1333, 800),
        pts_scale_ratio=1,
        flip=False,
        transforms=[
            dict(
                type='GlobalRotScaleTrans',
                rot_range=[0, 0],
                scale_ratio_range=[1.0, 1.0],
                translation_std=[0, 0, 0]),
            dict(
                type='RandomFlip3D',
                sync_2d=False,
                flip_ratio_bev_horizontal=0.5,
                flip_ratio_bev_vertical=0.5),
            dict(type='IndoorPointSample', num_points=100000),
            dict(
                type='DefaultFormatBundle3D',
                class_names=('table', 'chair', 'sofa', 'bookcase', 'board'),
                with_label=False),
            dict(type='Collect3D', keys=['points'])
        ])
]
data = dict(
    samples_per_gpu=8,
    workers_per_gpu=4,
    train=dict(
        type='RepeatDataset',
        times=13,
        dataset=dict(
            type='ConcatDataset',
            datasets=[
                dict(
                    type='S3DISDataset',
                    data_root='./data/s3dis/',
                    ann_file='./data/s3dis/s3dis_infos_Area_1.pkl',
                    pipeline=[
                        dict(
                            type='LoadPointsFromFile',
                            coord_type='DEPTH',
                            shift_height=False,
                            load_dim=6,
                            use_dim=[0, 1, 2, 3, 4, 5]),
                        dict(type='LoadAnnotations3D'),
                        dict(type='IndoorPointSample', num_points=100000),
                        dict(
                            type='RandomFlip3D',
                            sync_2d=False,
                            flip_ratio_bev_horizontal=0.5,
                            flip_ratio_bev_vertical=0.5),
                        dict(
                            type='GlobalRotScaleTrans',
                            rot_range=[-0.087266, 0.087266],
                            scale_ratio_range=[0.9, 1.1],
                            translation_std=[0.1, 0.1, 0.1],
                            shift_height=False),
                        dict(
                            type='DefaultFormatBundle3D',
                            class_names=('table', 'chair', 'sofa', 'bookcase',
                                         'board')),
                        dict(
                            type='Collect3D',
                            keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
                    ],
                    filter_empty_gt=True,
                    classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
                    box_type_3d='Depth'),
                dict(
                    type='S3DISDataset',
                    data_root='./data/s3dis/',
                    ann_file='./data/s3dis/s3dis_infos_Area_2.pkl',
                    pipeline=[
                        dict(
                            type='LoadPointsFromFile',
                            coord_type='DEPTH',
                            shift_height=False,
                            load_dim=6,
                            use_dim=[0, 1, 2, 3, 4, 5]),
                        dict(type='LoadAnnotations3D'),
                        dict(type='IndoorPointSample', num_points=100000),
                        dict(
                            type='RandomFlip3D',
                            sync_2d=False,
                            flip_ratio_bev_horizontal=0.5,
                            flip_ratio_bev_vertical=0.5),
                        dict(
                            type='GlobalRotScaleTrans',
                            rot_range=[-0.087266, 0.087266],
                            scale_ratio_range=[0.9, 1.1],
                            translation_std=[0.1, 0.1, 0.1],
                            shift_height=False),
                        dict(
                            type='DefaultFormatBundle3D',
                            class_names=('table', 'chair', 'sofa', 'bookcase',
                                         'board')),
                        dict(
                            type='Collect3D',
                            keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
                    ],
                    filter_empty_gt=True,
                    classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
                    box_type_3d='Depth'),
                dict(
                    type='S3DISDataset',
                    data_root='./data/s3dis/',
                    ann_file='./data/s3dis/s3dis_infos_Area_3.pkl',
                    pipeline=[
                        dict(
                            type='LoadPointsFromFile',
                            coord_type='DEPTH',
                            shift_height=False,
                            load_dim=6,
                            use_dim=[0, 1, 2, 3, 4, 5]),
                        dict(type='LoadAnnotations3D'),
                        dict(type='IndoorPointSample', num_points=100000),
                        dict(
                            type='RandomFlip3D',
                            sync_2d=False,
                            flip_ratio_bev_horizontal=0.5,
                            flip_ratio_bev_vertical=0.5),
                        dict(
                            type='GlobalRotScaleTrans',
                            rot_range=[-0.087266, 0.087266],
                            scale_ratio_range=[0.9, 1.1],
                            translation_std=[0.1, 0.1, 0.1],
                            shift_height=False),
                        dict(
                            type='DefaultFormatBundle3D',
                            class_names=('table', 'chair', 'sofa', 'bookcase',
                                         'board')),
                        dict(
                            type='Collect3D',
                            keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
                    ],
                    filter_empty_gt=True,
                    classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
                    box_type_3d='Depth'),
                dict(
                    type='S3DISDataset',
                    data_root='./data/s3dis/',
                    ann_file='./data/s3dis/s3dis_infos_Area_4.pkl',
                    pipeline=[
                        dict(
                            type='LoadPointsFromFile',
                            coord_type='DEPTH',
                            shift_height=False,
                            load_dim=6,
                            use_dim=[0, 1, 2, 3, 4, 5]),
                        dict(type='LoadAnnotations3D'),
                        dict(type='IndoorPointSample', num_points=100000),
                        dict(
                            type='RandomFlip3D',
                            sync_2d=False,
                            flip_ratio_bev_horizontal=0.5,
                            flip_ratio_bev_vertical=0.5),
                        dict(
                            type='GlobalRotScaleTrans',
                            rot_range=[-0.087266, 0.087266],
                            scale_ratio_range=[0.9, 1.1],
                            translation_std=[0.1, 0.1, 0.1],
                            shift_height=False),
                        dict(
                            type='DefaultFormatBundle3D',
                            class_names=('table', 'chair', 'sofa', 'bookcase',
                                         'board')),
                        dict(
                            type='Collect3D',
                            keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
                    ],
                    filter_empty_gt=True,
                    classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
                    box_type_3d='Depth'),
                dict(
                    type='S3DISDataset',
                    data_root='./data/s3dis/',
                    ann_file='./data/s3dis/s3dis_infos_Area_6.pkl',
                    pipeline=[
                        dict(
                            type='LoadPointsFromFile',
                            coord_type='DEPTH',
                            shift_height=False,
                            load_dim=6,
                            use_dim=[0, 1, 2, 3, 4, 5]),
                        dict(type='LoadAnnotations3D'),
                        dict(type='IndoorPointSample', num_points=100000),
                        dict(
                            type='RandomFlip3D',
                            sync_2d=False,
                            flip_ratio_bev_horizontal=0.5,
                            flip_ratio_bev_vertical=0.5),
                        dict(
                            type='GlobalRotScaleTrans',
                            rot_range=[-0.087266, 0.087266],
                            scale_ratio_range=[0.9, 1.1],
                            translation_std=[0.1, 0.1, 0.1],
                            shift_height=False),
                        dict(
                            type='DefaultFormatBundle3D',
                            class_names=('table', 'chair', 'sofa', 'bookcase',
                                         'board')),
                        dict(
                            type='Collect3D',
                            keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
                    ],
                    filter_empty_gt=True,
                    classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
                    box_type_3d='Depth')
            ],
            separate_eval=False)),
    val=dict(
        type='S3DISDataset',
        data_root='./data/s3dis/',
        ann_file='./data/s3dis/s3dis_infos_Area_5.pkl',
        pipeline=[
            dict(
                type='LoadPointsFromFile',
                coord_type='DEPTH',
                shift_height=False,
                load_dim=6,
                use_dim=[0, 1, 2, 3, 4, 5]),
            dict(
                type='MultiScaleFlipAug3D',
                img_scale=(1333, 800),
                pts_scale_ratio=1,
                flip=False,
                transforms=[
                    dict(
                        type='GlobalRotScaleTrans',
                        rot_range=[0, 0],
                        scale_ratio_range=[1.0, 1.0],
                        translation_std=[0, 0, 0]),
                    dict(
                        type='RandomFlip3D',
                        sync_2d=False,
                        flip_ratio_bev_horizontal=0.5,
                        flip_ratio_bev_vertical=0.5),
                    dict(type='IndoorPointSample', num_points=100000),
                    dict(
                        type='DefaultFormatBundle3D',
                        class_names=('table', 'chair', 'sofa', 'bookcase',
                                     'board'),
                        with_label=False),
                    dict(type='Collect3D', keys=['points'])
                ])
        ],
        classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
        test_mode=True,
        box_type_3d='Depth'),
    test=dict(
        type='S3DISDataset',
        data_root='./data/s3dis/',
        ann_file='./data/s3dis/s3dis_infos_Area_5.pkl',
        pipeline=[
            dict(
                type='LoadPointsFromFile',
                coord_type='DEPTH',
                shift_height=False,
                load_dim=6,
                use_dim=[0, 1, 2, 3, 4, 5]),
            dict(
                type='MultiScaleFlipAug3D',
                img_scale=(1333, 800),
                pts_scale_ratio=1,
                flip=False,
                transforms=[
                    dict(
                        type='GlobalRotScaleTrans',
                        rot_range=[0, 0],
                        scale_ratio_range=[1.0, 1.0],
                        translation_std=[0, 0, 0]),
                    dict(
                        type='RandomFlip3D',
                        sync_2d=False,
                        flip_ratio_bev_horizontal=0.5,
                        flip_ratio_bev_vertical=0.5),
                    dict(type='IndoorPointSample', num_points=100000),
                    dict(
                        type='DefaultFormatBundle3D',
                        class_names=('table', 'chair', 'sofa', 'bookcase',
                                     'board'),
                        with_label=False),
                    dict(type='Collect3D', keys=['points'])
                ])
        ],
        classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
        test_mode=True,
        box_type_3d='Depth'))
gpu_ids = range(0, 1)

2022-07-26 06:44:40,267 - mmdet - INFO - Set random seed to 0, deterministic: False
2022-07-26 06:44:41,884 - mmdet - INFO - Model:
SingleStageSparse3DDetector(
  (backbone): MEResNet3D(
    (conv1): Sequential(
      (0): MinkowskiConvolution(in=3, out=64, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
      (1): MinkowskiInstanceNorm(nchannels=64)
      (2): MinkowskiReLU()
      (3): MinkowskiMaxPooling(kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
    )
    (layer1): Sequential(
      (0): BasicBlock(
        (conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
        (downsample): Sequential(
          (0): MinkowskiConvolution(in=64, out=64, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
          (1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (2): BasicBlock(
        (conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
    )
    (layer2): Sequential(
      (0): BasicBlock(
        (conv1): MinkowskiConvolution(in=64, out=128, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
        (downsample): Sequential(
          (0): MinkowskiConvolution(in=64, out=128, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
          (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (2): BasicBlock(
        (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (3): BasicBlock(
        (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
    )
    (layer3): Sequential(
      (0): BasicBlock(
        (conv1): MinkowskiConvolution(in=128, out=256, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
        (downsample): Sequential(
          (0): MinkowskiConvolution(in=128, out=256, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
          (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (2): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (3): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (4): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (5): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
    )
    (layer4): Sequential(
      (0): BasicBlock(
        (conv1): MinkowskiConvolution(in=256, out=512, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
        (downsample): Sequential(
          (0): MinkowskiConvolution(in=256, out=512, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
          (1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
      (2): BasicBlock(
        (conv1): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
        (norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): MinkowskiReLU()
      )
    )
  )
  (neck_with_head): Fcaf3DNeckWithHead(
    (loss_centerness): CrossEntropyLoss()
    (loss_bbox): IoU3DLoss()
    (loss_cls): FocalLoss()
    (pruning): MinkowskiPruning()
    (out_block_0): Sequential(
      (0): MinkowskiConvolution(in=64, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
    )
    (up_block_1): Sequential(
      (0): MinkowskiGenerativeConvolutionTranspose(in=128, out=64, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
      (3): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (4): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): MinkowskiELU()
    )
    (out_block_1): Sequential(
      (0): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
    )
    (up_block_2): Sequential(
      (0): MinkowskiGenerativeConvolutionTranspose(in=256, out=128, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
      (3): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (4): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): MinkowskiELU()
    )
    (out_block_2): Sequential(
      (0): MinkowskiConvolution(in=256, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
    )
    (up_block_3): Sequential(
      (0): MinkowskiGenerativeConvolutionTranspose(in=512, out=256, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
      (3): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (4): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): MinkowskiELU()
    )
    (out_block_3): Sequential(
      (0): MinkowskiConvolution(in=512, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
      (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): MinkowskiELU()
    )
    (centerness_conv): MinkowskiConvolution(in=128, out=1, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
    (reg_conv): MinkowskiConvolution(in=128, out=6, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
    (cls_conv): MinkowskiConvolution(in=128, out=5, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
    (scales): ModuleList(
      (0): Scale()
      (1): Scale()
      (2): Scale()
      (3): Scale()
    )
  )
)
./data/s3dis/s3dis_infos_Area_1.pkl
./data/s3dis/s3dis_infos_Area_2.pkl
./data/s3dis/s3dis_infos_Area_3.pkl
./data/s3dis/s3dis_infos_Area_4.pkl
./data/s3dis/s3dis_infos_Area_6.pkl
./data/s3dis/s3dis_infos_Area_5.pkl
2022-07-26 06:44:45,311 - mmdet - INFO - Start running, host: root@adc44dc27719, work_dir: /mmdetection3d/work_dirs/fcaf3d_s3dis-3d-5class
2022-07-26 06:44:45,311 - mmdet - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) StepLrUpdaterHook                  
(NORMAL      ) CheckpointHook                     
(NORMAL      ) EvalHook                           
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) StepLrUpdaterHook                  
(NORMAL      ) EvalHook                           
(NORMAL      ) EmptyCacheHook                     
(LOW         ) IterTimerHook                      
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
before_train_iter:
(VERY_HIGH   ) StepLrUpdaterHook                  
(NORMAL      ) EvalHook                           
(LOW         ) IterTimerHook                      
 -------------------- 
after_train_iter:
(ABOVE_NORMAL) OptimizerHook                      
(NORMAL      ) CheckpointHook                     
(NORMAL      ) EvalHook                           
(NORMAL      ) EmptyCacheHook                     
(LOW         ) IterTimerHook                      
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
after_train_epoch:
(NORMAL      ) CheckpointHook                     
(NORMAL      ) EvalHook                           
(NORMAL      ) EmptyCacheHook                     
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
before_val_epoch:
(NORMAL      ) EmptyCacheHook                     
(LOW         ) IterTimerHook                      
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
before_val_iter:
(LOW         ) IterTimerHook                      
 -------------------- 
after_val_iter:
(NORMAL      ) EmptyCacheHook                     
(LOW         ) IterTimerHook                      
 -------------------- 
after_val_epoch:
(NORMAL      ) EmptyCacheHook                     
(VERY_LOW    ) TextLoggerHook                     
 -------------------- 
2022-07-26 06:44:45,311 - mmdet - INFO - workflow: [('train', 1)], max: 50 epochs
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [9,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [10,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [11,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [16,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [30,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "tools/train.py", line 223, in <module>
    main()
  File "tools/train.py", line 212, in main
    train_model(
  File "/mmdetection3d/mmdet3d/apis/train.py", line 27, in train_model
    train_detector(
  File "/opt/conda/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/opt/conda/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 237, in train_step
    losses = self(**data)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "/mmdetection3d/mmdet3d/models/detectors/base.py", line 58, in forward
    return self.forward_train(**kwargs)
  File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 48, in forward_train
    x = self.extract_feat(points, img_metas)
  File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 40, in extract_feat
    x = self.neck_with_head(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 102, in forward
    x = self._prune(x, scores)
  File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 124, in _prune
    prune_mask[permutation[mask]] = True
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1614378083779/work/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fb5e39412f2 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x5b (0x7fb5e393e67b in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x809 (0x7fb5e3b9a219 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7fb5e39293a4 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x6e0dda (0x7fb63a89ddda in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x6e0e71 (0x7fb63a89de71 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #26: __libc_start_main + 0xe7 (0x7fb675ef4bf7 in /lib/x86_64-linux-gnu/libc.so.6)
filaPro commented 2 years ago

Can you please try with our dockerfile or package versions from there, i.e. cuda 10.2? I also didn't try to train on a single gpu, only on 2 gpus. And one more thing, you will run out of memory for S3DIS dataset with your 16 Gb gpu, or you should decrease batch size. Btw, can you successfully run test.py on S3DIS with our pretrained model?

However anyway the error looks strange. I can have a look if nothing from above helps you...

week0425 commented 2 years ago

OK, I will try. I have tried to train S3DIS on the pointnet2. It worked well. So I think it may be not the environment's problem. I will give you a feedback when I find new messages. Thank you!

week0425 commented 2 years ago

Hi, @filaPro . I think I have found the reason, it's because the GPU doesn't have enough memory. So I will close this issue. Thanks again for your help.