Corpsecreate
commented
6 years ago 1. I actually had this already coded up, but didnt put it in the github release. I've now included it. The paper said there is 200 states because they disregard all situations where the player has a sum < 12, as the obvious action is to hit. This would reduce the complexity of the problem, and it is something I can add in but would prefer not to if possible. I also have the state where the player holds Blackjack (which is different to a soft 21), so all-in-all, I have approx 280 states. Easily changed to match the 200 if required.
2. There is an update in every step of the loop, so if the episode is 5 time steps, t will be compared with t+1 in the first loop, then t+1 with t+2, then t+2 with t+3 until the episode ends. Every state within the episode is getting an update, the network is actually updating multiple times within one episode. I'm considering changing this to a batch update after N number of episodes. Can easily be done but dont want to over-complicate anything until the logic of the update is sound and makes sense.
3. I think the logic of the loss function makes sense, but then maybe not? In theory, the output of the NN should be the expected value of an action, by calculating the difference between the predicted and the actual, that should be the adjustment required to the expected value. The learning rate dictates how far we move that expectation in each update. It also sounds like a SARSA update to me, but i might be getting my terminology mixed up. My understanding is that a SARSA update is when u only use a single sampled reward and then boostrap the maxQ of t+1 to perform an update. Its the special case where lambda = 1, with lambda = infinity being monte carlo. Will check your code in sec though to see how it differs.
Very much appreciate the time taken to read the code and provide feedback :)
edwbarker
I had a look at the code. Interesting! There’s a couple of things which could be causing problems:
The way you’ve encoded the state might make it quite difficult for the AI to figure out what’s going on. Particularly for the 5’s only problem. This is because it’s looking for a linear relationship between Q and each state, i.e. Q = ax_i. Have a look at how they encoded the state here: http://incompleteideas.net/book/bookdraft2017nov5.pdf (pages 129-130, or just ctrl-f “blackjack”). I think it will make it easier if each element of x is only ever 1 or 0 typically (not always, but typically);
I may have missed something, but the way you’ve coded the update to Q, it appears you skip every second step. You calculate x(t), Q1(t), then x(t+1), Q2(t+1), compare Q2 and Q1 (i.e. calculate loss), then generate x(t+2), Q1(t+2), compare x(t+3), Q2(t+3), but points in time t+1 and t+2 never get compared. This would break the chain it needs to predict the future. Again, not 100% clear, I may have missed something, but if you’re missing a step in the chain like this it will definitely break;
The loss function you defined should maybe work, however it’s not the most obvious / direct way to use the NN. See formula 16.3 from the link above, which is the more typical method for merging a NN and RL (at least for VF estimation). I think what you’ve done is called “residual” learning or something. Another example is the code I uploaded. I think technically you’re using Q-learning, not SARSA (not that it really matters). The NN code I generated uses SARSA.
Anyway, my best guess is that either 1 or 2 are making it fail to work. If you fix 1, I can try to run my NN code against it (I can run it already, but haven’t investigated it closely since I’m sceptical it will get decent results with the input defined the way it is at the moment, or at least not without a very big network and lots of training).