Custom dataset evaluate OOM

[GondorFu]

When I use this model on custom dataset, it is normal in training phase，but once in evaluation phase, it's always encountering out of memory for GPU.

What's the possible reason for this?

[Gofinge]

Hi, could you show me your custom dataset config (whole exported config in exp folder is better) and OOM output?

[GondorFu]

Hi, could you show me your custom dataset config (whole exported config in exp folder is better) and OOM output?

These are config and error log

Config: weight = None resume = False evaluate = True test_only = False seed = 318105 save_path = num_worker = 16 batch_size = 8 batch_size_val = None batch_size_test = 1 epoch = 100 eval_epoch = 100 save_freq = None eval_metric = 'mIoU' sync_bn = False enable_amp = True empty_cache = False find_unused_parameters = False max_batch_points = 100000000.0 mix_prob = 0.8 param_dicts = None test = dict(type='SegmentationTest') model = dict( type='ptv2m2', in_channels=6, num_classes= patch_embed_depth=2, patch_embed_channels=48, patch_embed_groups=6, patch_embed_neighbours=16, enc_depths=(2, 6, 2), enc_channels=(96, 192, 384), enc_groups=(12, 24, 48), enc_neighbours=(16, 16, 16), dec_depths=(1, 1, 1), dec_channels=(48, 96, 192), dec_groups=(6, 12, 24), dec_neighbours=(16, 16, 16), grid_sizes=(0.1, 0.2, 0.4), attn_qkv_bias=True, pe_multiplier=False, pe_bias=True, attn_drop_rate=0.0, drop_path_rate=0.3, enable_checkpoint=False, unpool_backend='interp') optimizer = dict(type='AdamW', lr=0.006, weight_decay=0.05) scheduler = dict(type='MultiStepLR', milestones=[0.6, 0.8], gamma=0.1) dataset_type = 'AutoScenesDataset' data_root = ' data = dict( num_classes= ignore_label=255, names=[' train=dict( type= split='train', data_root=' transform=[ dict(type='CenterShift', apply_z=True), dict(type='RandomScale', scale=[0.9, 1.1]), dict(type='RandomFlip', p=0.5), dict(type='RandomJitter', sigma=0.005, clip=0.02), dict(type='ChromaticAutoContrast', p=0.2, blend_factor=None), dict(type='ChromaticTranslation', p=0.95, ratio=0.05), dict(type='ChromaticJitter', p=0.95, std=0.05), dict( type='Voxelize', voxel_size=0.04, hash_type='fnv', mode='train', keys=('coord', 'color', 'label'), return_discrete_coord=True), dict(type='SphereCrop', point_max=100000, mode='random'), dict(type='CenterShift', apply_z=False), dict(type='NormalizeColor'), dict(type='ToTensor'), dict( type='Collect', keys=('coord', 'discrete_coord', 'label'), feat_keys=['coord', 'color']) ], test_mode=False, loop=1), val=dict( type=' split='val', data_root=' transform=[ dict(type='CenterShift', apply_z=True), dict( type='Copy', keys_dict=dict(coord='origin_coord', label='origin_label')), dict( type='Voxelize', voxel_size=0.04, hash_type='fnv', mode='train', keys=('coord', 'color', 'label'), return_discrete_coord=True), dict(type='CenterShift', apply_z=False), dict(type='NormalizeColor'), dict(type='ToTensor'), dict( type='Collect', keys=('coord', 'discrete_coord', 'label'), offset_keys_dict=dict(offset='coord'), feat_keys=['coord', 'color']) ], test_mode=False), test=dict( type=' split='test', data_root=' transform=[ dict(type='CenterShift', apply_z=True), dict(type='NormalizeColor') ], test_mode=True, test_cfg=dict( voxelize=dict( type='Voxelize', voxel_size=0.04, hash_type='fnv', mode='test', keys=('coord', 'color'), return_discrete_coord=True), crop=None, post_transform=[ dict(type='CenterShift', apply_z=False), dict(type='ToTensor'), dict( type='Collect', keys=('coord', 'discrete_coord', 'index'), feat_keys=('coord', 'color')) ], aug_transform=[[{ 'type': 'RandomScale', 'scale': [0.9, 0.9] }], [{ 'type': 'RandomScale', 'scale': [0.95, 0.95] }], [{ 'type': 'RandomScale', 'scale': [1, 1] }], [{ 'type': 'RandomScale', 'scale': [1.05, 1.05] }], [{ 'type': 'RandomScale', 'scale': [1.1, 1.1] }], [{ 'type': 'RandomScale', 'scale': [0.9, 0.9] }, { 'type': 'RandomFlip', 'p': 1 }], [{ 'type': 'RandomScale', 'scale': [0.95, 0.95] }, { 'type': 'RandomFlip', 'p': 1 }], [{ 'type': 'RandomScale', 'scale': [1, 1] }, { 'type': 'RandomFlip', 'p': 1 }], [{ 'type': 'RandomScale', 'scale': [1.05, 1.05] }, { 'type': 'RandomFlip', 'p': 1 }], [{ 'type': 'RandomScale', 'scale': [1.1, 1.1] }, { 'type': 'RandomFlip', 'p': 1 }]]))) criteria = [dict(type='CrossEntropyLoss', loss_weight=1.0, ignore_index=255)] num_worker_per_gpu = 2 batch_size_per_gpu = 1 batch_size_val_per_gpu = 1

Start Evaluation >>>>>>>>>>>>>>>> python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown len(cache)) python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown len(cache)) python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown len(cache)) Traceback (most recent call last): File "tools/train.py", line 34, in main() File "tools/train.py", line 29, in main cfg=(cfg,), File "PointTransformerV2/pcr/engines/launch.py", line 84, in launch daemon=False, File python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes while not context.join(): File "python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 3 terminated with the following error: Traceback (most recent call last): File "python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, args) File "PointTransformerV2/pcr/engines/launch.py", line 183, in _distributed_worker main_func(cfg) File "PointTransformerV2/tools/train.py", line 16, in main_worker trainer.train() File "PointTransformerV2/pcr/engines/defaults.py", line 216, in train self.after_epoch() File "PointTransformerV2/pcr/engines/defaults.py", line 321, in after_epoch self.eval() File "PointTransformerV2/pcr/engines/defaults.py", line 334, in eval output = self.model(input_dict) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward output = self.module(inputs[0], kwargs[0]) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 517, in forward points = self.patch_embed(points) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 411, in forward return self.blocks([coord, feat, offset]) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, *kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 209, in forward points = block(points, reference_index) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 157, in forward if not self.enable_checkpoint else checkpoint(self.attn, feat, coord, reference_index) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 108, in forward peb = self.linear_p_bias(pos) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, *kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "python3.7/site-packages/torch/nn/modules/container.py", line 141, in forward input = module(input) File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(*args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "PointTransformerV2/pcr/models/point_transformer2/point_transformer_v2m2_base.py", line 37, in forward return self.norm(input.transpose(1, 2).contiguous()).transpose(1, 2).contiguous() File "python3.7/site-packages/sampler/utils/wrapper.py", line 21, in run res = func(args, kwargs) File "python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "python3.7/site-packages/torch/nn/modules/batchnorm.py", line 179, in forward self.eps, File "python3.7/site-packages/torch/nn/functional.py", line 2283, in batch_norm input, weight, bias, running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: CUDA out of memory. Tried to allocate 5.38 GiB (GPU 3; 31.75 GiB total capacity; 28.67 GiB already allocated; 1.47 GiB free; 28.80 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Gofinge]

The config looks good. Is there any potential that the validation point clouds are significantly larger than the training point clouds? What about data_dict["coord"].shape? (It would be helpful if you can log it out before OOM)

[GondorFu]

I random split the train and val, so there won't be much difference. You can see before OOM, the code try to allocate 5.38 GB, in my custom dataset, each pc only about 100M and just one for each GPU, what do you think is the reason why it need to allocate such a large amount of memory?

[Gofinge]

I am sorry about that issue. I never encounter a similar problem. I notice that the validation batch size per GPU is identical to the train batch size per GPU (both 1). The memory consumption of the evaluation process should be much lower than the training process.

For debugging this issue, my suggestion is to print out the input shape before feeding it into the model.

[GondorFu]

I am sorry about that issue. I never encounter a similar problem. I notice that the validation batch size per GPU is identical to the train batch size per GPU (both 1). The memory consumption of the evaluation process should be much lower than the training process.

For debugging this issue, my suggestion is to print out the input shape before feeding it into the model.

this is the eval size

Start Evaluation >>>>>>>>>>>>>>>> val size >>>>>>>>>>>>>>>: 655712 val size >>>>>>>>>>>>>>>: 872887 val size >>>>>>>>>>>>>>>: 871667 val size >>>>>>>>>>>>>>>: 1273970 val size >>>>>>>>>>>>>>>: 1541887 val size >>>>>>>>>>>>>>>: 1918415 val size >>>>>>>>>>>>>>>: 1695826 val size >>>>>>>>>>>>>>>: 2831842

[Gofinge]

Hi, that was quite a huge number for a point cloud after voxelization. Maybe you can further validate whether the validation point cloud voxelized successfully.