Closed EvanHan09 closed 4 years ago
请问，楼主有没有遇到过在训练时python run_cnn.py train 开始后，只训练计算得到1个epoch 结果，就停止训练了？ 我检查了显卡的显存占用，发现没有出现内存泄露问题。继而又尝试了两种显存的分配方式，①分配了0.4的显存 ②自动适应分配。得到的结果和上面一样，均只训练一个epoch就停止了。 Configuring TensorBoard and Saver... Loading training and validation data... Time usage: 0:00:11 2019-06-03 11:40:30.224462: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties: name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.71 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 4.89GiB 2019-06-03 11:40:30.237900: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0 2019-06-03 11:40:30.996786: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-06-03 11:40:31.005045: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2019-06-03 11:40:31.010727: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2019-06-03 11:40:31.015885: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2457 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5) Training and evaluating... Epoch: 1 Iter: 0, Train Loss: 2.3, Train Acc: 10.94%, Val Loss: 2.3, Val Acc: 10.02%, Time: 0:00:02 * 能给解答一下吗？
Configuring TensorBoard and Saver... Loading training and validation data... Time usage: 0:00:11 2019-06-03 11:40:30.224462: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties: name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.71 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 4.89GiB 2019-06-03 11:40:30.237900: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0 2019-06-03 11:40:30.996786: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-06-03 11:40:31.005045: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2019-06-03 11:40:31.010727: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2019-06-03 11:40:31.015885: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2457 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5) Training and evaluating... Epoch: 1 Iter: 0, Train Loss: 2.3, Train Acc: 10.94%, Val Loss: 2.3, Val Acc: 10.02%, Time: 0:00:02 *
请问，楼主有没有遇到过在训练时python run_cnn.py train 开始后，只训练计算得到1个epoch 结果，就停止训练了？ 我检查了显卡的显存占用，发现没有出现内存泄露问题。继而又尝试了两种显存的分配方式，①分配了0.4的显存 ②自动适应分配。得到的结果和上面一样，均只训练一个epoch就停止了。
Configuring TensorBoard and Saver... Loading training and validation data... Time usage: 0:00:11 2019-06-03 11:40:30.224462: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties: name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.71 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 4.89GiB 2019-06-03 11:40:30.237900: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0 2019-06-03 11:40:30.996786: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-06-03 11:40:31.005045: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2019-06-03 11:40:31.010727: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2019-06-03 11:40:31.015885: I c:\users\user\source\repos\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2457 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5) Training and evaluating... Epoch: 1 Iter: 0, Train Loss: 2.3, Train Acc: 10.94%, Val Loss: 2.3, Val Acc: 10.02%, Time: 0:00:02 *能给解答一下吗？