yuval-alaluf
commented
3 years ago You opened another issue with the same error. I am closing this issue as I answered in your other issue.
Closed MetiCodePrivateLimited closed 3 years ago
yuval-alaluf
commented
3 years ago You opened another issue with the same error. I am closing this issue as I answered in your other issue.
I am trying to execute following command:
python3 scripts/train.py \ --dataset_type=ffhq_aging \ --exp_dir=/path/to/experiment \ --workers=2 \ --batch_size=2 \ --test_batch_size=2 \ --test_workers=2 \ --val_interval=2500 \ --save_interval=10000 \ --start_from_encoded_w_plus \ --id_lambda=0.1 \ --lpips_lambda=0.1 \ --lpips_lambda_aging=0.1 \ --lpips_lambda_crop=0.6 \ --l2_lambda=0.25 \ --l2_lambda_aging=0.25 \ --l2_lambda_crop=1 \ --w_norm_lambda=0.005 \ --aging_lambda=5 \ --cycle_lambda=1 \ --input_nc=4 \ --target_age=uniform_random \ --use_weighted_id_loss \ --checkpoint_path=/usr/local/SAM/trained/sam_ffhq_aging.pt
and getting following error:
Loading SAM from checkpoint: /usr/local/SAM/trained/sam_ffhq_aging.pt Loading ResNet ArcFace Loading dataset for ffhq_aging Number of training samples: 11 Number of test samples: 1 2021-07-14 11:50:06.332677: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 /root/anaconda3/envs/sam_envs/lib/python3.6/site-packages/torch/nn/functional.py:3121: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)) /usr/local/SAM/criteria/aging_loss.py:24: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. predict_age_pb = F.softmax(age_pb) Traceback (most recent call last): File "scripts/train.py", line 30, in
main()
File "scripts/train.py", line 26, in main
coach.train()
File "/usr/local/SAM/training/coach_aging.py", line 120, in train
y_recovered, latent_cycle = self.perform_forward_pass(y_hat_inverse)
File "/usr/local/SAM/training/coach_aging.py", line 78, in perform_forward_pass
y_hat, latent = self.net.forward(x, return_latents=True)
File "/usr/local/SAM/models/psp.py", line 90, in forward
return_latents=return_latents)
File "/root/anaconda3/envs/sam_envs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, kwargs)
File "/usr/local/SAM/models/stylegan2/model.py", line 529, in forward
out = conv1(out, latent[:, i], noise=noise1)
File "/root/anaconda3/envs/sam_envs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, *kwargs)
File "/usr/local/SAM/models/stylegan2/model.py", line 332, in forward
out = self.conv(input, style)
File "/root/anaconda3/envs/sam_envs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(input, kwargs)
File "/usr/local/SAM/models/stylegan2/model.py", line 257, in forward
out = self.blur(out)
File "/root/anaconda3/envs/sam_envs/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/SAM/models/stylegan2/model.py", line 85, in forward
out = upfirdn2d(input, self.kernel, pad=self.pad)
File "/usr/local/SAM/models/stylegan2/op/upfirdn2d.py", line 144, in upfirdn2d
input, kernel, (up, up), (down, down), (pad[0], pad[1], pad[0], pad[1])
File "/usr/local/SAM/models/stylegan2/op/upfirdn2d.py", line 116, in forward
input, kernel, up_x, up_y, down_x, down_y, pad_x0, pad_x1, pad_y0, pad_y1
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 7.93 GiB total capacity; 6.95 GiB already allocated; 192.44 MiB free; 7.13 GiB reserved in total by PyTorch)