search

BlueAmulet / BasicSR

Fork of Basic Super-Resolution codes for development. Includes ESRGAN, SFT-GAN for training and testing.
Apache License 2.0
18 stars 1 forks source link

'not enough memory'error after backup a training state #10

Open satoshils opened 3 years ago

satoshils commented 3 years ago

My PC has 32G RAM,and they were only used 40% while training. But after backup a training state,the log says RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 24774144 bytes. Buy new RAM! I already set n_workers: 1 and batch_size: 1,the result is that the training tried to allocate only 24774144 bytes. That's less than 25MB,and my PC has 32GB RAM,why it's not enough? MY CPU: AMD RYZEN 3700x (8cores) GPU: Geforce 1660super CUDA: cuda_11.0.2_451.48_win10 pytorch: torch-1.6.0-cp38-cp38-win_amd64

This is the log:

export CUDA_VISIBLE_DEVICES=0 20-08-11 07:32:47.805 - INFO: name: debug_newtest use_tb_logger: True model: sr scale: 4 gpu_ids: [0] datasets:[ train:[ name: DIV2K mode: LRHR dataroot_HR: ./data_samples/div2k/div2k_train_hr dataroot_LR: ./data_samples/div2k/DIV2K_train_LR subset_file: None use_shuffle: True n_workers: 1 batch_size: 1 HR_size: 64 use_flip: True use_rot: True phase: train scale: 4 data_type: img LR_nc: 3 HR_nc: 3 ] val:[ name: val_set5 mode: LRHR dataroot_HR: ./data_samples/div2k/div2k_valid_hr dataroot_LR: ./data_samples/div2k/DIV2K_valid_LR phase: val scale: 4 data_type: img LR_nc: 3 HR_nc: 3 ] ] path:[ root: ./ pretrain_model_G: ./experiments/pretrained_models/4x_ArtStation1337_FatalityMKII90000G_05_rebout_02.pth experiments_root: ./experiments\debug_newtest models: ./experiments\debug_newtest\models training_state: ./experiments\debug_newtest\training_state log: ./experiments\debug_newtest val_images: ./experiments\debug_newtest\val_images ] network_G:[ which_model_G: RRDB_net norm_type: None mode: CNA nf: 64 nb: 23 in_nc: 3 out_nc: 3 gc: 32 group: 1 scale: 4 ] train:[ lr_G: 0.0002 lr_scheme: MultiStepLR lr_steps: [200000, 400000, 600000, 800000] lr_gamma: 0.5 pixel_criterion: l1 pixel_weight: 1 val_freq: 8 manual_seed: 0 niter: 1000000 lr_decay_iter: 10 ] logger:[ print_freq: 2 save_checkpoint_freq: 8 backup_freq: 2 ] is_train: True batch_multiplier: 1

20-08-11 07:32:47.805 - INFO: Random seed: 0 20-08-11 07:32:47.815 - INFO: Dataset [LRHRDataset - DIV2K] is created. 20-08-11 07:32:47.815 - INFO: Number of train images: 800, iters: 800 20-08-11 07:32:47.815 - INFO: Total epochs needed: 1250 for iters 1,000,000 20-08-11 07:32:47.817 - INFO: Dataset [LRHRDataset - val_set5] is created. 20-08-11 07:32:47.817 - INFO: Number of val images in [val_set5]: 100 20-08-11 07:32:47.946 - INFO: Initialization method [kaiming] 20-08-11 07:32:49.037 - INFO: Loading pretrained model for G [./experiments/pretrained_models/4x_ArtStation1337_FatalityMKII90000G_05_rebout_02.pth] ... 20-08-11 07:32:49.205 - INFO: Remove frequency separation. 20-08-11 07:32:49.205 - INFO: Remove feature loss. 20-08-11 07:32:49.206 - INFO: Remove HFEN loss. 20-08-11 07:32:49.207 - INFO: Remove TV loss. 20-08-11 07:32:49.207 - INFO: Remove SSIM loss. 20-08-11 07:32:49.207 - INFO: Remove LPIPS loss. 20-08-11 07:32:49.207 - INFO: Remove GAN loss. 20-08-11 07:32:49.211 - INFO: Model [SRRaGANModel] is created. 20-08-11 07:32:49.211 - INFO: Start training from epoch: 0, iter: 0 20-08-11 07:32:51.501 - INFO: <epoch: 0, iter: 2, lr:2.000e-04> l_g_pix: 7.1235e-02 20-08-11 07:32:51.796 - INFO: Backup models and training states saved. 20-08-11 07:32:52.356 - INFO: <epoch: 0, iter: 4, lr:2.000e-04> l_g_pix: 5.8562e-02 20-08-11 07:32:52.600 - INFO: Backup models and training states saved. 20-08-11 07:32:53.110 - INFO: <epoch: 0, iter: 6, lr:2.000e-04> l_g_pix: 4.0749e-02 20-08-11 07:32:53.568 - INFO: Backup models and training states saved. 20-08-11 07:32:54.078 - INFO: <epoch: 0, iter: 8, lr:2.000e-04> l_g_pix: 2.3442e-02 20-08-11 07:32:54.280 - INFO: Models and training states saved. 20-08-11 07:32:54.576 - INFO: Backup models and training states saved. Setting up Perceptual loss... Loading model from: D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\lpips_weights\v0.1\squeeze.pth ...[net-lin [squeeze]] initialized ...Done Traceback (most recent call last): File "./codes/train.py", line 252, in main() File "./codes/train.py", line 213, in main avg_lpips += lpips.calculate_lpips(cropped_sr_img, cropped_gt_img) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\compute_dists.py", line 33, in calculate_lpips dist01 = model.forward(img2,img1) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\perceptual_loss.py", line 39, in forward return self.model.forward(target, pred) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\dist_model.py", line 116, in forward return self.net.forward(in0, in1, retPerLayer=retPerLayer) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\networks_basic.py", line 67, in forward feats0[kk], feats1[kk] = util.normalize_tensor(outs0[kk]), util.normalize_tensor(outs1[kk]) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\LPIPS\perceptual_loss.py", line 42, in normalize_tensor norm_factor = torch.sqrt(torch.sum(in_feat**2,dim=1,keepdim=True)) RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 24774144 bytes. Buy new RAM!

satoshils commented 3 years ago

my yaml file is

name: debug_newtest use_tb_logger: true model: sr scale: 4 gpu_ids:

satoshils commented 3 years ago

last time trainings Traceback

Traceback (most recent call last): File "./codes/train.py", line 252, in main() File "./codes/train.py", line 133, in main model.optimize_parameters(train_gen, current_step) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\SRRaGAN_model.py", line 347, in optimize_parameters self.fake_H = self.netG(self.var_L) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, kwargs) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\parallel\data_parallel.py", line 153, in forward return self.module(*inputs[0], *kwargs[0]) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "D:\FUN\GAME\3ds\Texture\ESRGAN\train_AI\BasicSR-lite\codes\models\modules\architectures\RRDBNet_arch.py", line 46, in forward x = self.model(x) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, *kwargs) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\container.py", line 117, in forward input = module(input) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(input, **kwargs) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\conv.py", line 419, in forward return self._conv_forward(input, self.weight) File "C:\Users\Satoshi\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\conv.py", line 415, in _conv_forward return F.conv2d(input, weight, self.bias, self.stride, RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (3 x 3). Kernel size can't be greater than actual input size