SARSA

[mattsqerror]

To implement SARSA with experience replay:

• The memory module should not compute "targets" or TD error. Memory should just store state/action/reward/next state information, and provide it in batches upon request.
• The memory module should have a provide a large number of options for how batches are returned. For the case of SARSA, state/action/reward/next state/next state action can be returned. For q-learning, just state/action/reward/next state is required
• It might be more efficient and easier in some ways to store memory in separate state, action, reward vectors.
• "Targets" should be computed by the models themselves, perhaps calling q-learning / sarsa classes to help compute.
• Policy should not be used to compute TD error. Policy and td-error are unrelated. Given enough time, you could compute an optimal Q-function using a completely random policy.

[EderSantana]

It might be more efficient and easier in some ways to store memory in separate state, action, reward vectors.

What do you mean by that? There is one memory that is a list of state-actions-reward information. Why would be better to split that and how to sync them up?

"Targets" should be computed by the models themselves, perhaps calling q-learning / sarsa classes to help compute.

Aren't they already? https://github.com/EderSantana/X/blob/master/x/memory.py#L120-L121 Also we can always use the memory callback for a functional modification on the go: https://github.com/EderSantana/X/blob/master/x/memory.py#L118