Closed vova0108 closed 4 years ago
It seems the dataset has problems.
You may check your dataset.
Could you provide more details of your config yml file?
Yes of course. No problem.
# general settings
name: wowa
model_type: ESRGANModel
scale: 4
num_gpu: 1
# dataset and data loader settings
datasets:
train:
name: DIV2K
type: PairedImageDataset
dataroot_gt: ./datasets/DIV2K_train_HR_sub
dataroot_lq: ./datasets/DIV2K_train_LR_bicubicX4_sub
io_backend:
type: disk
#server_list_cfg: /mnt/lustre/share/memcached_client/server_list.conf
#client_cfg: /mnt/lustre/share/memcached_client/client.conf
#sys_path: /mnt/lustre/share/pymc/py3
gt_size: 128
use_flip: true
use_rot: true
# data loader
use_shuffle: true
num_worker: 6 # per GPU
batch_size: 16
dataset_enlarge_ratio: 1000
val:
name: val_set14
type: PairedImageDataset
dataroot_gt: ./datasets/val_set14/Set14
dataroot_lq: ./datasets/val_set14/Set14_bicLRx4
io_backend:
type: disk
# network structures
network_g:
type: RRDBNet
num_in_ch: 3
num_out_ch: 3
num_feat: 64
num_block: 23
network_d:
type: VGGStyleDiscriminator128
num_in_ch: 3
num_feat: 64
# path
path:
pretrain_model_g: ./experiments/pretrained_models/ESRGAN_PSNR_SRx4_DF2K_official-150ff491.pth
strict_load: true
resume_state: ~
# training settings
train:
optim_g:
type: Adam
lr: !!float 1e-4
weight_decay: 0
betas: [0.9, 0.99]
optim_d:
type: Adam
lr: !!float 1e-4
weight_decay: 0
betas: [0.9, 0.99]
scheduler:
type: MultiStepLR
milestones: [50000, 100000, 200000, 300000]
gamma: 0.5
niter: 400000
warmup_iter: -1 # no warm up
# losses
pixel_opt:
type: L1Loss
loss_weight: !!float 1e-2
reduction: mean
perceptual_opt:
type: PerceptualLoss
layer_weights:
'conv5_4': 1 # before relu
vgg_type: vgg19
use_input_norm: true
perceptual_weight: 1.0
style_weight: 0
norm_img: false
criterion: l1
gan_opt:
type: GANLoss
gan_type: vanilla
real_label_val: 1.0
fake_label_val: 0.0
loss_weight: !!float 5e-3
net_d_iters: 1
net_d_init_iters: 0
manual_seed: 0
# validation settings
val:
val_freq: !!float 5e3
save_img: true
# logging settings
logger:
print_freq: 100
save_checkpoint_freq: !!float 1e3
use_tb_logger: true
# dist training settings
dist_params:
backend: nccl
port: 29748
It seems OK from the config file.
You may check the dataset. A simple way is to write a simple script to real all data to see whether there are any corrupted images~
Got it. I will try to recreate the dataset. I take DIV2K as a basis and independently reduce the resolution by 4 times.
Rebuilt the dataset, training now goes without errors. Thanks for the help!
I met the same error. But when i rebuild the datasets, the error still occur. I always resume the training when the eroor occur.... can you give me some other advice to solve this error?
After 2700 iterations, the process stops with an error.