I am trying to train it on Jetson nano - 4GB memory.

[hbzhang]

There, I am trying to train it on Jetson nano - 4GB memory.

Is this possible?

Can I reduce the resources requested?

Thanks!

[D-X-Y]

Which dataset are you trying to train?

[hbzhang]

I do not have good machine.

I just want to run very basic CIFAR trying to figure out this codeset.

Also, can I run it across a few nanos like 20 in parallel?

Thanks!!

[D-X-Y]

4 GB should be ok to run CIFAR. If not, you could decrease init_channels or layers in https://github.com/D-X-Y/GDAS/blob/master/scripts-cnn/train-cifar.sh#L36. The codes support GPU parallel.

[hbzhang]

Thanks. Tried

--init_channels 1 --layers 2 \

does not work

.....

Train model from scratch without pre-trained model or snapshot

==>>[2019-08-12-16:41:37] [Epoch=000/600] [Need: 00:00:00] LR=0.0250 ~ 0.0250, Batch=96 THCudaCheck FAIL file=/media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THCUNN/generic/SpatialAveragePooling.cu line=184 error=7 : too many resources requested for launch Traceback (most recent call last): File "./exps-cnn/train_base.py", line 89, in main() File "./exps-cnn/train_base.py", line 84, in main main_procedure(config, args.dataset, args.data_path, args, genotype, args.init_channels, args.layers, None, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 96, in main_procedure train_acc1, train_acc5, train_los = _train(train_loader, model, criterion, optimizer, 'train', epoch, config, args.print_freq, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 148, in _train loss.backward() File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuda runtime error (7) : too many resources requested for launch at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THCUNN/generic/SpatialAveragePooling.cu:184

Tried --init_channels 1 --layers 1 \

Does not work either ...

Train model from scratch without pre-trained model or snapshot

==>>[2019-08-12-16:43:46] [Epoch=000/600] [Need: 00:00:00] LR=0.0250 ~ 0.0250, Batch=96 Traceback (most recent call last): File "./exps-cnn/train_base.py", line 89, in main() File "./exps-cnn/train_base.py", line 84, in main main_procedure(config, args.dataset, args.data_path, args, genotype, args.init_channels, args.layers, None, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 96, in main_procedure train_acc1, train_acc5, train_los = _train(train_loader, model, criterion, optimizer, 'train', epoch, config, args.print_freq, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 138, in _train logits, logits_aux = model(inputs) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/nvidia/ros/nas/GDAS/lib/nas/CifarNet.py", line 81, in forward logits_aux = self.auxiliary_head(s1) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/nvidia/ros/nas/GDAS/lib/nas/CifarNet.py", line 24, in forward x = self.classifier(x.view(x.size(0),-1)) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, **kwargs) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward return F.linear(input, self.weight, self.bias) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear ret = torch.addmm(bias, input, weight.t()) RuntimeError: size mismatch, m1: [96 x 6912], m2: [768 x 10] at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMathBlas.cu:268

[D-X-Y]

Please try with init_channels >= 2 and layers >= 2

[hbzhang]

With this setting

init_channels = 2 and layers = 2

It complains the followings:

==>>[2019-08-12-18:51:54] [Epoch=000/600] [Need: 00:00:00] LR=0.0250 ~ 0.0250, Batch=96 THCudaCheck FAIL file=/media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THCUNN/generic/SpatialAveragePooling.cu line=184 error=7 : too many resources requested for launch Traceback (most recent call last): File "./exps-cnn/train_base.py", line 89, in main() File "./exps-cnn/train_base.py", line 84, in main main_procedure(config, args.dataset, args.data_path, args, genotype, args.init_channels, args.layers, None, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 96, in main_procedure train_acc1, train_acc5, train_los = _train(train_loader, model, criterion, optimizer, 'train', epoch, config, args.print_freq, log) File "/home/nvidia/ros/nas/GDAS/exps-cnn/train_utils.py", line 148, in _train loss.backward() File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/nvidia/.virtualenvs/py36/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuda runtime error (7) : too many resources requested for launch at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THCUNN/generic/SpatialAveragePooling.cu:184

[D-X-Y]

It seems a hardware problem. I can run successfully on my GPU.

[hbzhang]

So are you able to try the 4GB GPU memory situation with your GPU?

Which file(s) I need to look into to debug the resources issue?

I think Jetson may can run it in parallel... so how to make it run in a parallel (cluster) fashion?

Thanks!

[D-X-Y]

I did not have a 4 GPU memory situation, but I checked the GPU memory usage, which is lower than 4 GPU. The codes (by default) use parallel. Regarding the resources issue, I'm not familiar with how to debug that.

[hbzhang]

Thanks. I can look into it. There might be some differences between Jetson nano and typical GPU in hardware architecture. But it is certainly interesting to compare....

[D-X-Y]

No worries. sorry, I'm not familiar with Jetson nano. I would temporarily close this issue, and please feel free to reopen it if you want.