Memory allocation problem

[YeSho-cpp]

Hello, sorry to bother you, I am running a nuclear data set with maskdino, but my problem now is insufficient memory, my bathsize is changed to 2, numworkers is changed to 0, and I started running, but the efficiency is too slow, numworkers will report memory allocation failure even if it is changed to 1. I have two a6000 graphics cards, but they cannot be distributed and used at the same time, otherwise the memory can not be allocated. I would like to ask you which parameters should be modified to reduce the use of memory.

[YeSho-cpp]

This is my data set information

[10/18 11:09:21] d2.data.datasets.coco INFO: Loading /share/home/ncu10/Code/AI/Point_label/PointWSSIS/cell_data_root/coco/annotations/instances_train2017.json takes 2.70 seconds. [10/18 11:09:21] d2.data.datasets.coco INFO: Loaded 432 images in COCO format from /share/home/ncu10/Code/AI/Point_label/PointWSSIS/cell_data_root/coco/annotations/instances_train2017.json [10/18 11:09:21] d2.data.build INFO: Removed 0 images with no usable annotations. 432 images left. [10/18 11:09:21] d2.data.build INFO: Distribution of instances among all 80 categories: 	category	#instances	category	#instances	category	#instances
person	17073	bicycle	0	car	0
motorcycle	0	airplane	0	bus	0
train	0	truck	0	boat	0
traffic light	0	fire hydrant	0	stop sign	0
parking meter	0	bench	0	bird	0
cat	0	dog	0	horse	0
sheep	0	cow	0	elephant	0
bear	0	zebra	0	giraffe	0
backpack	0	umbrella	0	handbag	0
tie	0	suitcase	0	frisbee	0
skis	0	snowboard	0	sports ball	0
kite	0	baseball bat	0	baseball gl..	0
skateboard	0	surfboard	0	tennis racket	0
bottle	0	wine glass	0	cup	0
fork	0	knife	0	spoon	0
bowl	0	banana	0	apple	0
sandwich	0	orange	0	broccoli	0
carrot	0	hot dog	0	pizza	0
donut	0	cake	0	chair	0
couch	0	potted plant	0	bed	0
dining table	0	toilet	0	tv	0
laptop	0	mouse	0	remote	0
keyboard	0	cell phone	0	microwave	0
oven	0	toaster	0	sink	0
refrigerator	0	book	0	clock	0
vase	0	scissors	0	teddy bear	0
hair drier	0	toothbrush	0
total	17073					

[10/18 11:09:21] d2.data.build INFO: Using training sampler TrainingSampler [10/18 11:09:21] d2.data.common INFO: Serializing the dataset using: <class 'detectron2.data.common._TorchSerializedList'> [10/18 11:09:21] d2.data.common INFO: Serializing 432 elements to byte tensors and concatenating them all ... [10/18 11:09:22] d2.data.common INFO: Serialized dataset takes 28.01 MiB

[YeSho-cpp]

Using the resnet50 model [10/18 11:09:13] detectron2 INFO: Rank of current process: 0. World size: 1 [10/18 11:09:14] detectron2 INFO: Environment info:

---

sys.platform linux Python 3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0] numpy 1.24.4 detectron2 0.6 @/share/home/ncu10/Code/AI/Point_label/MaskDINO/detectron2/detectron2 Compiler GCC 9.4 CUDA compiler CUDA 11.4 detectron2 arch flags 8.6 DETECTRON2_ENV_MODULE PyTorch 1.13.1 @/share/home/ncu10/miniconda3/envs/py38/lib/python3.8/site-packages/torch PyTorch debug build False torch._C._GLIBCXX_USE_CXX11_ABI False GPU available Yes GPU 0 NVIDIA RTX A6000 (arch=8.6) Driver version 470.86 CUDA_HOME /share/home/ncu10/CUDA/CUDA11.4 Pillow 9.5.0 torchvision 0.14.1 @/share/home/ncu10/miniconda3/envs/py38/lib/python3.8/site-packages/torchvision torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6 fvcore 0.1.5.post20221221 iopath 0.1.9 cv2 4.8.0

CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file /share/home/ncu10/Code/AI/Point_label/MaskDINO/configs/coco/instance-segmentation/maskdino_R50_bs16_50ep_3s.yaml MODEL.WEIGHTS /share/home/ncu10/Code/AI/Point_label/MaskDINO/model_file/maskdino_r50_50ep_300q_hid1024_3sd1_instance_maskenhanced_mask46.1ap_box51.5ap.pth

[sym330]

same error

[FengLi-ust]

Sorry for the late reply. How much memory do you need in our case? We use about 30G for Resnet50 batch size 4.