RuntimeError: CUDA error: too many resources requested for launch

[maaaxac]

hi @dusty-nv , in trying to run train_ssd.py with the open images (python3 open_images_downloader.py --max-images=500 --class-names "Apple,Orange,Banana,Strawberry,Grape,Pear,Pineapple,Watermelon" --data=data/fruit)

this is the output i get, can you tell whats wrong with it? thanks in advance

python3 train_ssd.py --data=data/fruit --model-dir=models/fruit --batch-size=1 --num-workers=1 --epochs=1 2024-04-15 10:05:38 - Using CUDA... 2024-04-15 10:05:38 - Namespace(balance_data=False, base_net=None, base_net_lr=0.001, batch_size=1, checkpoint_folder='models/fruit', dataset_type='open_images', datasets=['data/fruit'], debug_steps=10, extra_layers_lr=None, freeze_base_net=False, freeze_net=False, gamma=0.1, log_level='info', lr=0.01, mb2_width_mult=1.0, milestones='80,100', momentum=0.9, net='mb1-ssd', num_epochs=1, num_workers=1, pretrained_ssd='models/mobilenet-v1-ssd-mp-0_675.pth', resolution=300, resume=None, scheduler='cosine', t_max=100, use_cuda=True, validation_epochs=1, validation_mean_ap=False, weight_decay=0.0005) 2024-04-15 10:06:45 - model resolution 300x300 2024-04-15 10:06:45 - SSDSpec(feature_map_size=19, shrinkage=16, box_sizes=SSDBoxSizes(min=60, max=105), aspect_ratios=[2, 3]) 2024-04-15 10:06:45 - SSDSpec(feature_map_size=10, shrinkage=32, box_sizes=SSDBoxSizes(min=105, max=150), aspect_ratios=[2, 3]) 2024-04-15 10:06:45 - SSDSpec(feature_map_size=5, shrinkage=64, box_sizes=SSDBoxSizes(min=150, max=195), aspect_ratios=[2, 3]) 2024-04-15 10:06:45 - SSDSpec(feature_map_size=3, shrinkage=100, box_sizes=SSDBoxSizes(min=195, max=240), aspect_ratios=[2, 3]) 2024-04-15 10:06:45 - SSDSpec(feature_map_size=2, shrinkage=150, box_sizes=SSDBoxSizes(min=240, max=285), aspect_ratios=[2, 3]) 2024-04-15 10:06:45 - SSDSpec(feature_map_size=1, shrinkage=300, box_sizes=SSDBoxSizes(min=285, max=330), aspect_ratios=[2, 3]) 2024-04-15 10:06:51 - Prepare training datasets. 2024-04-15 10:06:51 - loading annotations from: data/fruit/sub-train-annotations-bbox.csv 2024-04-15 10:06:52 - annotations loaded from: data/fruit/sub-train-annotations-bbox.csv num images: 404 2024-04-15 10:06:54 - Dataset Summary:Number of Images: 404 Minimum Number of Images for a Class: -1 Label Distribution: Apple: 261 Banana: 113 Grape: 136 Orange: 599 Pear: 191 Pineapple: 47 Strawberry: 550 Watermelon: 50 2024-04-15 10:06:54 - Stored labels into file models/fruit/labels.txt. 2024-04-15 10:06:54 - Train dataset size: 404 2024-04-15 10:06:54 - Prepare Validation datasets. 2024-04-15 10:06:54 - loading annotations from: data/fruit/sub-test-annotations-bbox.csv 2024-04-15 10:06:54 - annotations loaded from: data/fruit/sub-test-annotations-bbox.csv num images: 73 2024-04-15 10:06:55 - Dataset Summary:Number of Images: 73 Minimum Number of Images for a Class: -1 Label Distribution: Apple: 11 Banana: 9 Grape: 21 Orange: 62 Pear: 6 Pineapple: 10 Strawberry: 73 Watermelon: 11 2024-04-15 10:06:55 - Validation dataset size: 73 2024-04-15 10:06:55 - Build network. 2024-04-15 10:06:58 - Init from pretrained SSD models/mobilenet-v1-ssd-mp-0_675.pth 2024-04-15 10:07:01 - Took 2.97 seconds to load the model. 2024-04-15 10:07:02 - Learning rate: 0.01, Base net learning rate: 0.001, Extra Layers learning rate: 0.01. 2024-04-15 10:07:02 - Uses CosineAnnealingLR scheduler. 2024-04-15 10:07:02 - Start training from epoch 0. /usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead. warnings.warn(warning.format(ret)) Traceback (most recent call last): File "train_ssd.py", line 406, in train(train_loader, net, criterion, optimizer, device=DEVICE, debug_steps=args.debug_steps, epoch=epoch) File "train_ssd.py", line 149, in train loss.backward() File "/usr/local/lib/python3.6/dist-packages/torch/_tensor.py", line 255, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 149, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: CUDA error: too many resources requested for launch CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.