Closed havietisov closed 3 years ago
You aren't calling the learn function in the main loop.
I'll be doing a video on actor critic in tensorflow 2 this upcoming week, you should watch it and check it out. It's a much cleaner implementation.
Didn't mean to close it before giving the OP time to comment.
Damn, that was embarassing and stupid of me, i'm sorry. However, it still failing to improve even after 180+ episodes. Here is the log (source file has been updated) :
episode 167 score -245.29 average score -263.41
episode 168 score -579.41 average score -269.46
episode 169 score -349.60 average score -269.98
episode 170 score -194.49 average score -268.06
episode 171 score -276.85 average score -268.63
episode 172 score -272.52 average score -268.43
episode 173 score -238.40 average score -268.99
episode 174 score -306.81 average score -268.89
episode 175 score -152.71 average score -268.69
episode 176 score -410.33 average score -270.20
episode 177 score -314.56 average score -272.95
episode 178 score -281.38 average score -273.94
episode 179 score -236.19 average score -274.66
episode 180 score -381.23 average score -275.26
episode 181 score -292.67 average score -276.26
episode 182 score -124.96 average score -274.34
episode 183 score -222.39 average score -274.91
episode 184 score -221.86 average score -274.15
episode 185 score -390.36 average score -276.50
episode 186 score -468.62 average score -279.71
episode 187 score -336.55 average score -279.89
episode 188 score -290.84 average score -279.67`
Same happened when i tried some of your code (https://github.com/philtabor/Actor-Critic-Methods-Paper-To-Code/tree/master/ActorCritic).
I literally copied it and had to change nothing. Log is following :
episode 44 score -523.9 average score -231.1
episode 45 score -106.3 average score -228.4
episode 46 score -233.1 average score -228.5
episode 47 score -124.3 average score -226.3
episode 48 score -86.0 average score -223.5
episode 49 score -221.1 average score -223.4
episode 50 score -421.5 average score -227.3
episode 51 score -522.7 average score -233.0
episode 52 score -351.9 average score -235.2
episode 53 score -176.0 average score -234.1
episode 54 score -414.9 average score -237.4
episode 55 score -125.2 average score -235.4
episode 56 score -273.4 average score -236.1
episode 57 score -96.5 average score -233.7
episode 58 score -540.9 average score -238.9
episode 59 score -122.6 average score -236.9
episode 60 score -186.6 average score -236.1
episode 61 score -263.3 average score -236.6
episode 62 score -272.6 average score -237.1
episode 63 score -23.6 average score -233.8
episode 64 score -163.9 average score -232.7
episode 65 score -419.4 average score -235.5
episode 66 score -247.5 average score -235.7
episode 67 score -220.6 average score -235.5
episode 68 score -518.1 average score -239.6
episode 69 score -161.8 average score -238.5
episode 70 score -71.6 average score -236.1
episode 71 score -8.5 average score -233.0
episode 72 score -198.3 average score -232.5
episode 73 score -337.2 average score -233.9
episode 74 score -245.1 average score -234.1
episode 75 score -615.1 average score -239.1
episode 76 score -401.5 average score -241.2
episode 77 score -289.3 average score -241.8
episode 78 score -321.4 average score -242.8
episode 79 score -207.5 average score -242.4
episode 80 score -50.3 average score -240.0
episode 81 score -483.8 average score -243.0
episode 82 score -248.1 average score -243.0
episode 83 score -178.2 average score -242.3
episode 84 score -194.9 average score -241.7
episode 85 score -253.1 average score -241.8
episode 86 score -235.4 average score -241.8
episode 87 score -373.9 average score -243.3
episode 88 score -280.5 average score -243.7
episode 89 score -312.7 average score -244.4
episode 90 score -127.4 average score -243.2
episode 91 score -186.4 average score -242.5
episode 92 score -179.2 average score -241.9
episode 93 score -234.0 average score -241.8
episode 94 score -170.7 average score -241.0
episode 95 score -455.6 average score -243.3
episode 96 score -68.8 average score -241.5
episode 97 score -243.6 average score -241.5
episode 98 score -173.7 average score -240.8
episode 99 score -328.9 average score -241.7
episode 100 score -222.0 average score -242.5
episode 101 score -390.5 average score -242.1
episode 102 score -267.5 average score -243.2
episode 103 score -54.5 average score -243.1
episode 104 score -176.3 average score -243.9
episode 105 score -170.9 average score -242.1
episode 106 score -169.3 average score -242.9
episode 107 score -26.2 average score -239.8
episode 108 score -23.2 average score -237.2
episode 109 score -299.9 average score -237.0
episode 110 score -76.2 average score -235.9
episode 111 score -425.6 average score -239.4
episode 112 score -111.1 average score -238.6
episode 113 score -232.3 average score -237.5
episode 114 score -80.1 average score -237.1
episode 115 score -495.7 average score -240.9
episode 116 score -118.2 average score -238.5
episode 117 score -231.8 average score -238.4
episode 118 score 5.2 average score -237.2
episode 119 score -375.7 average score -238.3
episode 120 score -151.8 average score -239.0
episode 121 score 28.0 average score -235.9
episode 122 score -264.3 average score -237.5
episode 123 score -250.5 average score -236.8
episode 124 score -21.3 average score -234.7
episode 125 score -279.2 average score -235.2
episode 126 score -447.0 average score -235.6
episode 127 score -326.3 average score -234.4
episode 128 score -235.5 average score -236.0
episode 129 score -305.7 average score -237.0
episode 130 score -172.0 average score -237.7
episode 131 score -21.8 average score -237.5`
Do you find that result surprising? If so, why?
Well, speaking of the last link i've referenced, i don't know what to expect, i just assumed that it's supposed to work. Last piece of log doesn't show significant improvement over 150 or so episodes in this environment, and even if i keep it for longer, i'm starting to see quite opposite effect, with score sometimes going below thousand. However. Whenever someone's following the tutorial, i think it's fair for that person to expect that tutorial will lead you to something functional, unless it's a tutorial on "here's why this stuff doesn't work".
In the video tutorial for to this code, right at the very beginning of the video... the first 10 seconds or so, I show a learning plot.
What does this plot show?
Well, i removed any parts that used your utility class after copying the code. I suspect you want me to actually put it back, and to see the graph.
No, I want you to actually watch the video and tell me what the learning plot tells you about how long it takes the agent to learn.
Yep, you're right, my bad, i didn't paid attention to the actual numbers on Game axis. Now i need to fix my CUDA installation because even on a threadripper each iteration is too slow. Thank you for your feedback and for your patience.
Hi. I was following your youtube tutorial on Actor-critic method in continious space (lunar lander). However, despite having same code, my model almost never score higher than zero, nevermind reaching anywhere near 200, even after significant amount of episodes. Code is following : https://github.com/6opoDuJIo/RL_Playground/blob/master/lunar_lander.py And part of the log file is :