search

RozDavid / LanguageGroundedSemseg

Implementation for ECCV 2022 paper Language-Grounded Indoor 3D Semantic Segmentation in the Wild
98 stars 14 forks source link

TypeError: max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ), but expected one of: #1

Closed Cemm23333 closed 2 years ago

Cemm23333 commented 2 years ago

Dear RozDavid: Thank you for your kindly sharing codes. I encountered some issues when I ran . scripts/train_scannet_slurm.sh (lg_semseg) root@Id3a3e2672004018f9:/hy-tmp/LanguageGroundedSemseg/downstream/insseg# . scripts/train_scannet_slurm.sh [2022-08-22 16:07:37,376][root][INFO] - ===> Configurations [2022-08-22 16:07:37,376][root][INFO] - {'net': {'model': 'Res16UNet34D', 'conv1_kernel_size': 3, 'weights': '/hy-tmp/Data/ScanNet/weights/34D_CLIP_pretrain.ckpt', 'weights_for_inner_model': False, 'dilations': [1, 1, 1, 1], 'wrapper_type': None, 'wrapper_region_type': 1, 'wrapper_kernel_size': 3, 'wrapper_lr': 0.1, 'meanfield_iterations': 10, 'crf_spatial_sigma': 1, 'crf_chromatic_sigma': 12}, 'optimizer': {'optimizer': 'SGD', 'lr': 0.02, 'sgd_momentum': 0.9, 'sgd_dampening': 0.1, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'weight_decay': 0.0001, 'param_histogram_freq': 100, 'save_param_histogram': False, 'iter_size': 1, 'bn_momentum': 0.02, 'loss_type': 'cross_entropy', 'scheduler': 'MultiStepLR', 'max_iter': 600000, 'max_epoch': 400, 'step_size': 20000.0, 'step_gamma': 0.3, 'poly_power': 0.9, 'exp_gamma': 0.95, 'exp_step_size': 445, 'multi_step_milestones': [150, 200]}, 'data': {'dataset': 'Scannet200Voxelization2cmDataset', 'train_file': None, 'voxel_size': 0.05, 'data_dir': 'data', 'sampled_inds': None, 'temporal_dilation': 30, 'temporal_numseq': 3, 'point_lim': -1, 'pre_point_lim': -1, 'batch_size': 2, 'val_batch_size': 1, 'test_batch_size': 1, 'cache_data': False, 'num_workers': 4, 'ignore_label': -1, 'return_transformation': True, 'ignore_duplicate_class': False, 'partial_crop': 0, 'train_limit_numpoints': 0, 'synthia_path': '/home/chrischoy/datasets/Synthia/Synthia4D', 'synthia_camera_path': '/home/chrischoy/datasets/Synthia/%s/CameraParams/', 'synthia_camera_intrinsic_file': 'intrinsics.txt', 'synthia_camera_extrinsics_file': 'Stereo_Right/Omni_F/%s.txt', 'temporal_rand_dilation': False, 'temporal_rand_numseq': False, 'scannet_path': '/hy-tmp/Datasets/limited/scannet_200_insseg', 'stanford3d_path': '/home/chrischoy/datasets/Stanford3D', 'category_weights': 'feature_data/scannet200_category_weights.pkl'}, 'train': {'is_train': True, 'stat_freq': 20, 'val_freq': 1000, 'empty_cache_freq': 1, 'train_phase': 'train', 'val_phase': 'val', 'overwrite_weights': True, 'resume': '', 'resume_optimizer': 'True,', 'eval_upsample': False, 'lenient_weight_loading': True}, 'distributed': {'distributed_world_size': 8, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': None, 'distributed_port': 10010, 'device_id': 0, 'distributed_no_spawn': True, 'ddp_backend': 'c10d', 'bucket_cap_mb': 25}, 'augmentation': {'use_feat_aug': True, 'data_aug_color_trans_ratio': 0.1, 'data_aug_color_jitter_std': 0.05, 'normalize_color': True, 'data_aug_scale_min': 0.9, 'data_aug_scale_max': 1.1, 'data_aug_hue_max': 0.5, 'data_aug_saturation_max': 0.2}, 'test': {'test_phase': 'test', 'test_stat_freq': 100, 'evaluate_benchmark': False, 'dual_set_cluster': False, 'visualize': False, 'visualize_path': './visualize'}, 'misc': {'is_cuda': True, 'load_path': None, 'log_step': 50, 'log_level': 'INFO', 'num_gpus': 1, 'seed': 123, 'log_dir': '/hy-tmp/Data/outputs/Scannet200Voxelization2cmDataset/Res16UNet34D-', 'load_bn': 'all_bn', 'resume_config': None, 'train_stuff': False, 'wandb_id': ''}} [2022-08-22 16:07:37,377][root][INFO] - ===> Initializing dataloader [2022-08-22 16:07:37,378][root][INFO] - Loading Scannet200Voxelization2cmDataset: scannetv2_train.txt [2022-08-22 16:07:37,393][root][INFO] - ===> Building model building model, 3 [2022-08-22 16:07:38,222][root][INFO] - ===> Number of trainable parameters: Res16UNet34D: 79695019 [2022-08-22 16:07:38,222][root][INFO] - Res16UNet34D( (conv0p1s1): MinkowskiConvolution(in=3, out=32, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (bn0): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (conv1p1s2): MinkowskiConvolution(in=32, out=32, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block1): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv2p2s2): MinkowskiConvolution(in=32, out=32, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block2): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=32, out=64, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv3p4s2): MinkowskiConvolution(in=64, out=64, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn3): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block3): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=64, out=128, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv4p8s2): MinkowskiConvolution(in=128, out=128, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn4): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block4): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=256, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (4): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (5): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr4p16s2): MinkowskiConvolutionTranspose(in=256, out=256, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr4): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block5): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=384, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=384, out=256, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr5p8s2): MinkowskiConvolutionTranspose(in=256, out=256, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr5): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block6): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=320, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=320, out=256, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr6p4s2): MinkowskiConvolutionTranspose(in=256, out=256, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr6): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block7): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=288, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=288, out=256, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr7p2s2): MinkowskiConvolutionTranspose(in=256, out=512, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr7): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block8): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=544, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=544, out=512, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (final): MinkowskiConvolution(in=512, out=200, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (relu): MinkowskiReLU() (offsets_pre): MinkowskiConvolution(in=512, out=512, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (bntr_offset): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (offsets): MinkowskiConvolution(in=512, out=3, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) ) [2022-08-22 16:07:38,227][root][INFO] - ===> Loading weights: /hy-tmp/Data/ScanNet/weights/34D_CLIP_pretrain.ckpt [2022-08-22 16:07:38,788][root][INFO] - Loading Pytorch-Lightning weights from state [2022-08-22 16:07:38,789][root][INFO] - Loading weights:conv0p1s1.kernel, bn0.bn.weight, bn0.bn.bias, bn0.bn.running_mean, bn0.bn.running_var, bn0.bn.num_batches_tracked, conv1p1s2.kernel, bn1.bn.weight, bn1.bn.bias, bn1.bn.running_mean, bn1.bn.running_var, bn1.bn.num_batches_tracked, block1.0.conv1.kernel, block1.0.norm1.bn.weight, block1.0.norm1.bn.bias, block1.0.norm1.bn.running_mean, block1.0.norm1.bn.running_var, block1.0.norm1.bn.num_batches_tracked, block1.0.conv2.kernel, block1.0.norm2.bn.weight, block1.0.norm2.bn.bias, block1.0.norm2.bn.running_mean, block1.0.norm2.bn.running_var, block1.0.norm2.bn.num_batches_tracked, block1.1.conv1.kernel, block1.1.norm1.bn.weight, block1.1.norm1.bn.bias, block1.1.norm1.bn.running_mean, block1.1.norm1.bn.running_var, block1.1.norm1.bn.num_batches_tracked, block1.1.conv2.kernel, block1.1.norm2.bn.weight, block1.1.norm2.bn.bias, block1.1.norm2.bn.running_mean, block1.1.norm2.bn.running_var, block1.1.norm2.bn.num_batches_tracked, conv2p2s2.kernel, bn2.bn.weight, bn2.bn.bias, bn2.bn.running_mean, bn2.bn.running_var, bn2.bn.num_batches_tracked, block2.0.conv1.kernel, block2.0.norm1.bn.weight, block2.0.norm1.bn.bias, block2.0.norm1.bn.running_mean, block2.0.norm1.bn.running_var, block2.0.norm1.bn.num_batches_tracked, block2.0.conv2.kernel, block2.0.norm2.bn.weight, block2.0.norm2.bn.bias, block2.0.norm2.bn.running_mean, block2.0.norm2.bn.running_var, block2.0.norm2.bn.num_batches_tracked, block2.0.downsample.0.kernel, block2.0.downsample.1.bn.weight, block2.0.downsample.1.bn.bias, block2.0.downsample.1.bn.running_mean, block2.0.downsample.1.bn.running_var, block2.0.downsample.1.bn.num_batches_tracked, block2.1.conv1.kernel, block2.1.norm1.bn.weight, block2.1.norm1.bn.bias, block2.1.norm1.bn.running_mean, block2.1.norm1.bn.running_var, block2.1.norm1.bn.num_batches_tracked, block2.1.conv2.kernel, block2.1.norm2.bn.weight, block2.1.norm2.bn.bias, block2.1.norm2.bn.running_mean, block2.1.norm2.bn.running_var, block2.1.norm2.bn.num_batches_tracked, block2.2.conv1.kernel, block2.2.norm1.bn.weight, block2.2.norm1.bn.bias, block2.2.norm1.bn.running_mean, block2.2.norm1.bn.running_var, block2.2.norm1.bn.num_batches_tracked, block2.2.conv2.kernel, block2.2.norm2.bn.weight, block2.2.norm2.bn.bias, block2.2.norm2.bn.running_mean, block2.2.norm2.bn.running_var, block2.2.norm2.bn.num_batches_tracked, conv3p4s2.kernel, bn3.bn.weight, bn3.bn.bias, bn3.bn.running_mean, bn3.bn.running_var, bn3.bn.num_batches_tracked, block3.0.conv1.kernel, block3.0.norm1.bn.weight, block3.0.norm1.bn.bias, block3.0.norm1.bn.running_mean, block3.0.norm1.bn.running_var, block3.0.norm1.bn.num_batches_tracked, block3.0.conv2.kernel, block3.0.norm2.bn.weight, block3.0.norm2.bn.bias, block3.0.norm2.bn.running_mean, block3.0.norm2.bn.running_var, block3.0.norm2.bn.num_batches_tracked, block3.0.downsample.0.kernel, block3.0.downsample.1.bn.weight, block3.0.downsample.1.bn.bias, block3.0.downsample.1.bn.running_mean, block3.0.downsample.1.bn.running_var, block3.0.downsample.1.bn.num_batches_tracked, block3.1.conv1.kernel, block3.1.norm1.bn.weight, block3.1.norm1.bn.bias, block3.1.norm1.bn.running_mean, block3.1.norm1.bn.running_var, block3.1.norm1.bn.num_batches_tracked, block3.1.conv2.kernel, block3.1.norm2.bn.weight, block3.1.norm2.bn.bias, block3.1.norm2.bn.running_mean, block3.1.norm2.bn.running_var, block3.1.norm2.bn.num_batches_tracked, block3.2.conv1.kernel, block3.2.norm1.bn.weight, block3.2.norm1.bn.bias, block3.2.norm1.bn.running_mean, block3.2.norm1.bn.running_var, block3.2.norm1.bn.num_batches_tracked, block3.2.conv2.kernel, block3.2.norm2.bn.weight, block3.2.norm2.bn.bias, block3.2.norm2.bn.running_mean, block3.2.norm2.bn.running_var, block3.2.norm2.bn.num_batches_tracked, block3.3.conv1.kernel, block3.3.norm1.bn.weight, block3.3.norm1.bn.bias, block3.3.norm1.bn.running_mean, block3.3.norm1.bn.running_var, block3.3.norm1.bn.num_batches_tracked, block3.3.conv2.kernel, block3.3.norm2.bn.weight, block3.3.norm2.bn.bias, block3.3.norm2.bn.running_mean, block3.3.norm2.bn.running_var, block3.3.norm2.bn.num_batches_tracked, conv4p8s2.kernel, bn4.bn.weight, bn4.bn.bias, bn4.bn.running_mean, bn4.bn.running_var, bn4.bn.num_batches_tracked, block4.0.conv1.kernel, block4.0.norm1.bn.weight, block4.0.norm1.bn.bias, block4.0.norm1.bn.running_mean, block4.0.norm1.bn.running_var, block4.0.norm1.bn.num_batches_tracked, block4.0.conv2.kernel, block4.0.norm2.bn.weight, block4.0.norm2.bn.bias, block4.0.norm2.bn.running_mean, block4.0.norm2.bn.running_var, block4.0.norm2.bn.num_batches_tracked, block4.0.downsample.0.kernel, block4.0.downsample.1.bn.weight, block4.0.downsample.1.bn.bias, block4.0.downsample.1.bn.running_mean, block4.0.downsample.1.bn.running_var, block4.0.downsample.1.bn.num_batches_tracked, block4.1.conv1.kernel, block4.1.norm1.bn.weight, block4.1.norm1.bn.bias, block4.1.norm1.bn.running_mean, block4.1.norm1.bn.running_var, block4.1.norm1.bn.num_batches_tracked, block4.1.conv2.kernel, block4.1.norm2.bn.weight, block4.1.norm2.bn.bias, block4.1.norm2.bn.running_mean, block4.1.norm2.bn.running_var, block4.1.norm2.bn.num_batches_tracked, block4.2.conv1.kernel, block4.2.norm1.bn.weight, block4.2.norm1.bn.bias, block4.2.norm1.bn.running_mean, block4.2.norm1.bn.running_var, block4.2.norm1.bn.num_batches_tracked, block4.2.conv2.kernel, block4.2.norm2.bn.weight, block4.2.norm2.bn.bias, block4.2.norm2.bn.running_mean, block4.2.norm2.bn.running_var, block4.2.norm2.bn.num_batches_tracked, block4.3.conv1.kernel, block4.3.norm1.bn.weight, block4.3.norm1.bn.bias, block4.3.norm1.bn.running_mean, block4.3.norm1.bn.running_var, block4.3.norm1.bn.num_batches_tracked, block4.3.conv2.kernel, block4.3.norm2.bn.weight, block4.3.norm2.bn.bias, block4.3.norm2.bn.running_mean, block4.3.norm2.bn.running_var, block4.3.norm2.bn.num_batches_tracked, block4.4.conv1.kernel, block4.4.norm1.bn.weight, block4.4.norm1.bn.bias, block4.4.norm1.bn.running_mean, block4.4.norm1.bn.running_var, block4.4.norm1.bn.num_batches_tracked, block4.4.conv2.kernel, block4.4.norm2.bn.weight, block4.4.norm2.bn.bias, block4.4.norm2.bn.running_mean, block4.4.norm2.bn.running_var, block4.4.norm2.bn.num_batches_tracked, block4.5.conv1.kernel, block4.5.norm1.bn.weight, block4.5.norm1.bn.bias, block4.5.norm1.bn.running_mean, block4.5.norm1.bn.running_var, block4.5.norm1.bn.num_batches_tracked, block4.5.conv2.kernel, block4.5.norm2.bn.weight, block4.5.norm2.bn.bias, block4.5.norm2.bn.running_mean, block4.5.norm2.bn.running_var, block4.5.norm2.bn.num_batches_tracked, convtr4p16s2.kernel, bntr4.bn.weight, bntr4.bn.bias, bntr4.bn.running_mean, bntr4.bn.running_var, bntr4.bn.num_batches_tracked, block5.0.conv1.kernel, block5.0.norm1.bn.weight, block5.0.norm1.bn.bias, block5.0.norm1.bn.running_mean, block5.0.norm1.bn.running_var, block5.0.norm1.bn.num_batches_tracked, block5.0.conv2.kernel, block5.0.norm2.bn.weight, block5.0.norm2.bn.bias, block5.0.norm2.bn.running_mean, block5.0.norm2.bn.running_var, block5.0.norm2.bn.num_batches_tracked, block5.0.downsample.0.kernel, block5.0.downsample.1.bn.weight, block5.0.downsample.1.bn.bias, block5.0.downsample.1.bn.running_mean, block5.0.downsample.1.bn.running_var, block5.0.downsample.1.bn.num_batches_tracked, block5.1.conv1.kernel, block5.1.norm1.bn.weight, block5.1.norm1.bn.bias, block5.1.norm1.bn.running_mean, block5.1.norm1.bn.running_var, block5.1.norm1.bn.num_batches_tracked, block5.1.conv2.kernel, block5.1.norm2.bn.weight, block5.1.norm2.bn.bias, block5.1.norm2.bn.running_mean, block5.1.norm2.bn.running_var, block5.1.norm2.bn.num_batches_tracked, convtr5p8s2.kernel, bntr5.bn.weight, bntr5.bn.bias, bntr5.bn.running_mean, bntr5.bn.running_var, bntr5.bn.num_batches_tracked, block6.0.conv1.kernel, block6.0.norm1.bn.weight, block6.0.norm1.bn.bias, block6.0.norm1.bn.running_mean, block6.0.norm1.bn.running_var, block6.0.norm1.bn.num_batches_tracked, block6.0.conv2.kernel, block6.0.norm2.bn.weight, block6.0.norm2.bn.bias, block6.0.norm2.bn.running_mean, block6.0.norm2.bn.running_var, block6.0.norm2.bn.num_batches_tracked, block6.0.downsample.0.kernel, block6.0.downsample.1.bn.weight, block6.0.downsample.1.bn.bias, block6.0.downsample.1.bn.running_mean, block6.0.downsample.1.bn.running_var, block6.0.downsample.1.bn.num_batches_tracked, block6.1.conv1.kernel, block6.1.norm1.bn.weight, block6.1.norm1.bn.bias, block6.1.norm1.bn.running_mean, block6.1.norm1.bn.running_var, block6.1.norm1.bn.num_batches_tracked, block6.1.conv2.kernel, block6.1.norm2.bn.weight, block6.1.norm2.bn.bias, block6.1.norm2.bn.running_mean, block6.1.norm2.bn.running_var, block6.1.norm2.bn.num_batches_tracked, convtr6p4s2.kernel, bntr6.bn.weight, bntr6.bn.bias, bntr6.bn.running_mean, bntr6.bn.running_var, bntr6.bn.num_batches_tracked, block7.0.conv1.kernel, block7.0.norm1.bn.weight, block7.0.norm1.bn.bias, block7.0.norm1.bn.running_mean, block7.0.norm1.bn.running_var, block7.0.norm1.bn.num_batches_tracked, block7.0.conv2.kernel, block7.0.norm2.bn.weight, block7.0.norm2.bn.bias, block7.0.norm2.bn.running_mean, block7.0.norm2.bn.running_var, block7.0.norm2.bn.num_batches_tracked, block7.0.downsample.0.kernel, block7.0.downsample.1.bn.weight, block7.0.downsample.1.bn.bias, block7.0.downsample.1.bn.running_mean, block7.0.downsample.1.bn.running_var, block7.0.downsample.1.bn.num_batches_tracked, block7.1.conv1.kernel, block7.1.norm1.bn.weight, block7.1.norm1.bn.bias, block7.1.norm1.bn.running_mean, block7.1.norm1.bn.running_var, block7.1.norm1.bn.num_batches_tracked, block7.1.conv2.kernel, block7.1.norm2.bn.weight, block7.1.norm2.bn.bias, block7.1.norm2.bn.running_mean, block7.1.norm2.bn.running_var, block7.1.norm2.bn.num_batches_tracked, convtr7p2s2.kernel, bntr7.bn.weight, bntr7.bn.bias, bntr7.bn.running_mean, bntr7.bn.running_var, bntr7.bn.num_batches_tracked, block8.0.conv1.kernel, block8.0.norm1.bn.weight, block8.0.norm1.bn.bias, block8.0.norm1.bn.running_mean, block8.0.norm1.bn.running_var, block8.0.norm1.bn.num_batches_tracked, block8.0.conv2.kernel, block8.0.norm2.bn.weight, block8.0.norm2.bn.bias, block8.0.norm2.bn.running_mean, block8.0.norm2.bn.running_var, block8.0.norm2.bn.num_batches_tracked, block8.0.downsample.0.kernel, block8.0.downsample.1.bn.weight, block8.0.downsample.1.bn.bias, block8.0.downsample.1.bn.running_mean, block8.0.downsample.1.bn.running_var, block8.0.downsample.1.bn.num_batches_tracked, block8.1.conv1.kernel, block8.1.norm1.bn.weight, block8.1.norm1.bn.bias, block8.1.norm1.bn.running_mean, block8.1.norm1.bn.running_var, block8.1.norm1.bn.num_batches_tracked, block8.1.conv2.kernel, block8.1.norm2.bn.weight, block8.1.norm2.bn.bias, block8.1.norm2.bn.running_mean, block8.1.norm2.bn.running_var, block8.1.norm2.bn.num_batches_tracked Global seed set to 123 /usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py:20: LightningDeprecationWarning: The pl.plugins.training_type.ddp.DDPPlugin is deprecated in v1.6 and will be removed in v1.8. Use pl.strategies.ddp.DDPStrategy instead. rank_zero_deprecation( GPU available: True, used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs Global seed set to 123 Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1 [2022-08-22 16:07:38,998][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 0 [2022-08-22 16:07:38,999][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for 1 nodes.

distributed_backend=nccl All distributed processes registered. Starting with 1 processes

Missing logger folder: /hy-tmp/Data/outputs/Scannet200Voxelization2cmDataset/Res16UNet34D-/lightning_logs LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

| Name | Type | Params

0 | model | Res16UNet34D | 79.7 M 1 | criterion | CrossEntropyLoss | 0
2 | scores | Precision | 0
3 | accuracy | Recall | 0
4 | semantic_loss | MetricAverageMeter | 0
5 | offset_dir_loss | MetricAverageMeter | 0
6 | offset_norm_loss | MetricAverageMeter | 0
7 | total_loss | MetricAverageMeter | 0
8 | iou_scores | JaccardIndex | 0
9 | average_precision | AveragePrecision | 0

79.7 M Trainable params 0 Non-trainable params 79.7 M Total params 318.780 Total estimated model params size (MB) [2022-08-22 16:07:43,135][root][INFO] - Loading Scannet200Voxelization2cmDataset: scannetv2_train.txt [2022-08-22 16:07:43,265][root][INFO] - Loading Scannet200Voxelization2cmDataset: scannetv2_val.txt Epoch 0: 0%| | 0/913 [00:00<?, ?it/s]Traceback (most recent call last): File "ddp_main.py", line 120, in main trainer.fit(pl_module, ckpt_path=config.train.resume) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit self._call_and_handle_interrupt( File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, kwargs) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch return function(*args, *kwargs) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run results = self._run_stage() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage return self._run_train() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train self.fit_loop.run() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(args, kwargs) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 266, in advance self._outputs = self.epoch_loop.run(self._data_fetcher) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, kwargs) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 171, in advance batch = next(data_fetcher) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in next return self.fetching_function() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 259, in fetching_function self._fetch_next_batch(self.dataloader_iter) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 273, in _fetch_next_batch batch = next(iterator) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 558, in next return self.request_next_batch(self.loader_iters) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 570, in request_next_batch return apply_to_collection(loader_iters, Iterator, next) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 99, in apply_to_collection return function(data, *args, *kwargs) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data return self._process_data(data) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data data.reraise() File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise raise self.exc_type(msg) TypeError: Caught TypeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/hy-tmp/LanguageGroundedSemseg/downstream/insseg/datasets/dataset.py", line 326, in getitem coords, feats, labels, instances = self.input_transform(coords, feats, labels, instances) File "/hy-tmp/LanguageGroundedSemseg/downstream/insseg/datasets/transforms.py", line 237, in call args = t(args) File "/hy-tmp/LanguageGroundedSemseg/downstream/insseg/datasets/transforms.py", line 178, in call coord_max = np.max(coords[:,curr_ax]) File "<__array_function__ internals>", line 180, in amax File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2791, in amax return _wrapreduction(a, np.maximum, 'max', axis, None, out, File "/usr/local/miniconda3/envs/lg_semseg/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 84, in _wrapreduction return reduction(axis=axis, out=out, passkwargs) TypeError: max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ), but expected one of:

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. Epoch 0: 0%|
Have you ever meet this problem? And how to fix it?Thank you very much

RozDavid commented 2 years ago

Hey @Cemm23333,

I haven't had this specific error, but based on your logs you are having a problem in the instance segmentation training augmentation scripts. There are two potential things could be uninitialized in your loader:

Both of them should be fairly easy to inspect further with a print statement directly before the problematic line. Let me know if you managed to solve it or have further information for debugging.

Cheers, David

Cemm23333 commented 2 years ago

Thank you very much. I solved this issue through replaceing coord_max = np.max(coords[:, curr_ax]) with coord_max = np.max(coords[:, curr_ax].detach().numpy()).And Could you give some instructions on evalution ?

RozDavid commented 2 years ago

I'm glad you managed to solve the problem! Both for semantic and instance segmentation the evaluation scripts are included in the validation phase. Quantitative results should be logged to Wandb and Tensorboard depending on your preference.

Also if you would like to also visualize the results you could enable that in the config parameters here. Let me know know if you have any more specific questions.

Closing the issue now as it's not realted anymore to the original topic, but feel free to open an other one with follow up questions.

Cheers, David