CUDA error:out of memory when test

[Lily1992]

when I run test.py use my dataset,CUDA error: out of memory , how to solve it. My GPU is GTX1080TI 11G,the batch is 1,and chunk size is [1].

[Duankaiwen]

Please show the full log

[Lily1992]

(venv) F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10>python demo.py C:\Users\pc\Anaconda3\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(flo at).type. from ._conv import register_converters as _register_converters loading from cache file: cache\coco_val2014.pkl loading annotations into memory... Done (t=0.22s) creating index... index created! module_file: models.CenterNet-104 total parameters: 209955020 loading from F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test..\cache\nnet\CenterNet-104\CenterNet-104_23000.pkl 0%| | 0/2 [00:00<?, ?it/s]C :\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead. warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))

Traceback (most recent call last): File "demo.py", line 15, in bboxes = detector(image) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\base.py", line 24, in call bboxes = self._inference(image, *args, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\base.py", line 20, in _inference return self._func(self._db, self._nnet, image.copy(), *args, *kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\centernet.py", line 113, in inference dets, center = decode_func(nnet, images, K, ae_threshold=ae_threshold, kernel=nms_kernel) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\centernet.py", line 57, in kp_decode detections, center = nnet.test([images], ae_threshold=ae_threshold, K=K, kernel=kernel) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\nnet\py_factory.py", line 125, in test return self.model(xs, kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\nnet\py_factory.py", line 43, in forward return self.module(*xs, *kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\py_utils\kp.py", line 301, in forward return self._test(*xs, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\py_utils\kp.py", line 280, in _test tl_cnv = tlcnv(cnv) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, *kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\CenterNet-104.py", line 61, in forward bn1 = self.bn1(conv1) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\batchnorm.py", line 76, in forward exponential_average_factor, self.eps) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1623, in batch_norm training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: CUDA out of memory. Tried to allocate 598.00 MiB (GPU 0; 11.00 GiB total capacity; 8.11 GiB already allocated; 523.63 MiB free; 10.58 MiB cached)

[Lily1992]

(venv) F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10>python test.py CenterNet-104 --suffix multi_scale C:\Users\pc\Anaconda3\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(flo at).type. from ._conv import register_converters as _register_converters cfg_file: config\CenterNet-104-multi_scale.json loading all datasets... 2 GPUs split: validation loading from cache file: cache\coco_val.pkl loading annotations into memory... Done (t=0.11s) creating index... index created! system config... {'batch_size': 2, 'cache_dir': 'cache', 'chunk_sizes': [1, 1], 'config_dir': 'config', 'data_dir': 'F:/DATAset', 'data_rng': <mtrand.RandomState object at 0x000001E4CEAEA798>, 'dataset': 'MSCOCO', 'decay_rate': 10, 'display': 100, 'learning_rate': 2.5e-05, 'max_iter': 50000, 'nnet_rng': <mtrand.RandomState object at 0x000001E4CEAEA7E0>, 'opt_algo': 'adam', 'prefetch_size': 10, 'pretrain': None, 'result_dir': 'results', 'sampling_function': 'kp_detection', 'snapshot': 1000, 'snapshot_name': 'CenterNet-104', 'stepsize': 45000, 'test_split': 'testing', 'train_split': 'training', 'val_iter': 100, 'val_split': 'validation', 'weight_decay': False, 'weight_decay_rate': 1e-05, 'weight_decay_type': 'l2'} db config... {'ae_threshold': 0.5, 'border': 128, 'categories': 10, 'data_aug': True, 'gaussian_bump': True, 'gaussian_iou': 0.7, 'gaussian_radius': -1, 'input_size': [511, 511], 'kp_categories': 1, 'lighting': True, 'max_per_image': 100, 'merge_bbox': True, 'nms_algorithm': 'exp_soft_nms', 'nms_kernel': 3, 'nms_threshold': 0.5, 'output_sizes': [[128, 128]], 'rand_color': True, 'rand_crop': True, 'rand_pushes': False, 'rand_samples': False, 'rand_scale_max': 1.4, 'rand_scale_min': 0.6, 'rand_scale_step': 0.1, 'rand_scales': array([0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3]), 'special_crop': False, 'test_scales': [0.6, 1, 1.2, 1.5, 1.8], 'top_k': 70, 'weight_exp': 10} loading parameters at iteration: 23000 building neural network... module_file: models.CenterNet-104 total parameters: 209955020 loading parameters... loading model from cache\nnet\CenterNet-104\CenterNet-104_23000.pkl locating kps: 0%| | 0/3444 [00:00<?, ?it/s]C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate ins tead. warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))

Traceback (most recent call last): File "test.py", line 113, in test(testing_db, args.split, args.testiter, args.debug, args.suffix) File "test.py", line 74, in test testing(db, nnet, result_dir, debug=debug) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\coco.py", line 321, in testing return globals()[system_configs.sampling_function](db, nnet, result_dir, debug=debug) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\coco.py", line 129, in kp_detection dets, center = decode_func(nnet, images, K, ae_threshold=ae_threshold, kernel=nms_kernel) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\test\coco.py", line 54, in kp_decode detections, center = nnet.test([images], ae_threshold=ae_threshold, K=K, kernel=kernel) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\nnet\py_factory.py", line 125, in test return self.model(*xs, kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, *kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\nnet\py_factory.py", line 43, in forward return self.module(xs, kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\py_utils\kp.py", line 301, in forward return self._test(*xs, *kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\py_utils\kp.py", line 280, in _test tl_cnv = tlcnv(cnv) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\CenterNet-104.py", line 64, in forward conv2 = self.conv2(relu1) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, *kwargs) File "F:\CL_wiscom\OneStage_objectDetection\CenterNet-CenterNet_pytorch_win10\models\py_utils\utils.py", line 25, in forward conv = self.conv(x) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, **kwargs) File "C:\Users\pc\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 320, in forward self.padding, self.dilation, self.groups) RuntimeError: CUDA out of memory. Tried to allocate 598.00 MiB (GPU 0; 11.00 GiB total capacity; 8.49 GiB already allocated; 157.88 MiB free; 9.73 MiB cached)

[Duankaiwen]

@Lily1992 Hi, multi_scale testing needs large GPU memory, Titan X works for it. Or you can reduce the 'test_scales' in config/CenterNet-xxx-multi_scale.json. For example, modify "test_scales": [0.6, 1, 1.2, 1.5].