I have download a keras sample program to test my computer capability with tensorflow-GPU, but it seems that the program loop will be slow down by hidden memory issue.
I have add time recorder to record the time used in each loop.
from time import time
.
.
.
start = time()
.
.
.
if frame_count % update_target_network == 0:
elapse = time()-start
print("Loop time is {}!!!".format(elapse))
start = time()
Expected behaviour
The time used for each loop should not increase as frame count increase.
Actual behaviour
I have star program at 16:41 and it reach 100000 frame count at 22:30, time used is at below:
10000 frame count: 381.39
20000 frame count: 701.10
30000 frame count: 1090.84
40000 frame count: 1490.53
50000 frame count: 1916.36
60000 frame count: 2280.29
70000 frame count: 2674.47
80000 frame count: 3248.64
90000 frame count: 3646.62
100000 frame count: 4091.70
Log
I don't know how to get the log, please tell me if you really need it.
Subject of the issue
I have download a keras sample program to test my computer capability with tensorflow-GPU, but it seems that the program loop will be slow down by hidden memory issue.
Your environment
tensorflow-GPU ver: v2.4 GPU: RTX-3090 CUDA: v11.4.2
Steps to reproduce
The program I used: https://keras.io/examples/rl/deep_q_network_breakout/
I have add time recorder to record the time used in each loop.
from time import time
. . .
start = time()
. . .
if frame_count % update_target_network == 0:
elapse = time()-start
print("Loop time is {}!!!".format(elapse))
start = time()
Expected behaviour
The time used for each loop should not increase as frame count increase.
Actual behaviour
I have star program at 16:41 and it reach 100000 frame count at 22:30, time used is at below: 10000 frame count: 381.39 20000 frame count: 701.10 30000 frame count: 1090.84 40000 frame count: 1490.53 50000 frame count: 1916.36 60000 frame count: 2280.29 70000 frame count: 2674.47 80000 frame count: 3248.64 90000 frame count: 3646.62 100000 frame count: 4091.70
Log