Closed TanLuuu closed 1 year ago
作者你好，我在训练你的网络的时候，出现了PSNR为-42，l_pix为nan的问题，无法正常训练，请问该怎么解决呢，我只是改了数据集的路径，其他都是保持不变的
2023-06-15 21:08:13,778 INFO: Model [ImageEventRestorationModel] is created. 2023-06-15 21:08:13,804 INFO: Resuming training from epoch: 0, iter: 648. 2023-06-15 21:08:13,951 INFO: Start training from epoch: 0, iter: 648 2023-06-15 21:08:18,450 INFO: [debug..][epoch: 0, iter: 649, lr:(2.000e-04,2.000e-05,)] [eta: 5 days, 8:38:06, time (data): 4.499 (1.987)] l_pix: nan 2023-06-15 21:08:19,032 INFO: [debug..][epoch: 0, iter: 650, lr:(2.000e-04,2.000e-05,)] [eta: 4 days, 0:29:47, time (data): 0.581 (0.003)] l_pix: nan 2023-06-15 21:08:19,602 INFO: [debug..][epoch: 0, iter: 651, lr:(2.000e-04,2.000e-05,)] [eta: 3 days, 8:16:14, time (data): 0.570 (0.003)] l_pix: nan 2023-06-15 21:08:20,166 INFO: [debug..][epoch: 0, iter: 652, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 22:27:08, time (data): 0.563 (0.002)] l_pix: nan 2023-06-15 21:08:20,727 INFO: [debug..][epoch: 0, iter: 653, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 15:53:43, time (data): 0.561 (0.002)] l_pix: nan 2023-06-15 21:08:21,331 INFO: [debug..][epoch: 0, iter: 654, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 11:32:32, time (data): 0.603 (0.002)] l_pix: nan 2023-06-15 21:08:21,946 INFO: [debug..][epoch: 0, iter: 655, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 8:21:24, time (data): 0.615 (0.002)] l_pix: nan 2023-06-15 21:08:22,558 INFO: [debug..][epoch: 0, iter: 656, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 5:51:33, time (data): 0.611 (0.002)] l_pix: nan 2023-06-15 21:08:22,558 INFO: Saving models and training states. 2023-06-15 21:26:51,089 INFO: Validation debug, # psnr: -42.1933 # ssim: 0.0002 2023-06-15 21:26:51,656 INFO: [debug..][epoch: 0, iter: 657, lr:(2.000e-04,2.000e-05,)] [eta: 257 days, 21:51:19, time (data): 0.563 (0.002)] l_pix: nan 2023-06-15 21:26:52,259 INFO: [debug..][epoch: 0, iter: 658, lr:(2.000e-04,2.000e-05,)] [eta: 234 days, 14:08:50, time (data): 0.602 (0.003)] l_pix: nan 2023-06-15 21:26:52,840 INFO: [debug..][epoch: 0, iter: 659, lr:(2.000e-04,2.000e-05,)] [eta: 215 days, 3:37:24, time (data): 0.581 (0.002)] l_pix: nan
作者你好，我在训练你的网络的时候，出现了PSNR为-42，l_pix为nan的问题，无法正常训练，请问该怎么解决呢，我只是改了数据集的路径，其他都是保持不变的
2023-06-15 21:08:13,778 INFO: Model [ImageEventRestorationModel] is created. 2023-06-15 21:08:13,804 INFO: Resuming training from epoch: 0, iter: 648. 2023-06-15 21:08:13,951 INFO: Start training from epoch: 0, iter: 648 2023-06-15 21:08:18,450 INFO: [debug..][epoch: 0, iter: 649, lr:(2.000e-04,2.000e-05,)] [eta: 5 days, 8:38:06, time (data): 4.499 (1.987)] l_pix: nan 2023-06-15 21:08:19,032 INFO: [debug..][epoch: 0, iter: 650, lr:(2.000e-04,2.000e-05,)] [eta: 4 days, 0:29:47, time (data): 0.581 (0.003)] l_pix: nan 2023-06-15 21:08:19,602 INFO: [debug..][epoch: 0, iter: 651, lr:(2.000e-04,2.000e-05,)] [eta: 3 days, 8:16:14, time (data): 0.570 (0.003)] l_pix: nan 2023-06-15 21:08:20,166 INFO: [debug..][epoch: 0, iter: 652, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 22:27:08, time (data): 0.563 (0.002)] l_pix: nan 2023-06-15 21:08:20,727 INFO: [debug..][epoch: 0, iter: 653, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 15:53:43, time (data): 0.561 (0.002)] l_pix: nan 2023-06-15 21:08:21,331 INFO: [debug..][epoch: 0, iter: 654, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 11:32:32, time (data): 0.603 (0.002)] l_pix: nan 2023-06-15 21:08:21,946 INFO: [debug..][epoch: 0, iter: 655, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 8:21:24, time (data): 0.615 (0.002)] l_pix: nan 2023-06-15 21:08:22,558 INFO: [debug..][epoch: 0, iter: 656, lr:(2.000e-04,2.000e-05,)] [eta: 2 days, 5:51:33, time (data): 0.611 (0.002)] l_pix: nan 2023-06-15 21:08:22,558 INFO: Saving models and training states. 2023-06-15 21:26:51,089 INFO: Validation debug, # psnr: -42.1933 # ssim: 0.0002 2023-06-15 21:26:51,656 INFO: [debug..][epoch: 0, iter: 657, lr:(2.000e-04,2.000e-05,)] [eta: 257 days, 21:51:19, time (data): 0.563 (0.002)] l_pix: nan 2023-06-15 21:26:52,259 INFO: [debug..][epoch: 0, iter: 658, lr:(2.000e-04,2.000e-05,)] [eta: 234 days, 14:08:50, time (data): 0.602 (0.003)] l_pix: nan 2023-06-15 21:26:52,840 INFO: [debug..][epoch: 0, iter: 659, lr:(2.000e-04,2.000e-05,)] [eta: 215 days, 3:37:24, time (data): 0.581 (0.002)] l_pix: nan