What does Lstm mean for this crate?

[dbsxdbsx]

I know Lstm is a variant of RNN in the context of neural network of machine learning. And in the original NEAT paper, the author did mention a kind of RNN, with which the game double pole is solved----the RNN here is not what we refer to in the context of machine learning, right?

But what does lstm mean in the context of NEAT for this crate? Does it mean that NEAT could generate lstm structure (like in this paper), and with which could solve non-Markov decision problem?

[pkalivas]

There are two concepts of RNN here. The first is the same as the original NEAT paper where certain neurons in the NEAT graph are recurrent. The second is the LSTM layer which evolves a traditional LSTM layer where each gate in the layer is a NEAT neural network graph.

The structure for the LSTM is defined already, the only evolution that takes place with that layer is on the internal gates. I would think it would be able to solve non-markov decision problems, however I haven't tested it extensively with those.

[dbsxdbsx]

@pkalivas, thanks for your reply. Indeed, I am also looking forward that NEAT with LSTM could solve non-Markov decision problems, though there is rare paper related to it, even in field of deep reinforcement learning.

Just a side question. In the original NEAT paper, if I were not wrong, the RNN mentioned there is just a simple edge(parameter weight) that connects to a node the edge initialed from. In this context, isn't it an endless loop?

[pkalivas]

I think in theory any sort of reinforcement LSTM implementation should be able to solve your problem. The actual architecture of the layer and it's internal gates can be debated but I would think (again, without explicitly testing) would yield the same results.

To address the second point, no, this wouldn't result in an infinite loop. The original paper gives a picture of a neuron with an edge connecting to itself. In this implementation I decided to avoid that to prevent weird internal loops within the graph in favor of allowing each neuron to be recurrent if the settings allowed. It does this by keeping track of the previous neuron's activated value and applying it to the current activated value. This prevents an extra edge in the graph.

[dbsxdbsx]

I am a little confused. Does the NEAT-RNN also has a concept called "hidden state", like that when talking about LSTM, GRU... ?

Let me make it clear. For instance, in the game double pole as represented in paper NEAT, there are many states s1,s2.... in an episode. If there are hidden states when using NEAT-RNN, then there is a state called h1 produced during state s1 by the NEAT-RNN unit within a network. And then h1 would be combined with s2 for afterward calculation within the NEAT-RNN unit.

[pkalivas]

In a way, yes. Any type of RNN will have a hidden state or some representation of it regardless of if it is NEAT or a traditional neural network. The LSTM, GRU, and normal NEAT keep track of states for memory purposes in this implementation.

[dbsxdbsx]

In a way, yes. Any type of RNN will have a hidden state or some representation of it regardless of if it is NEAT or a traditional neural network. The LSTM, GRU, and normal NEAT keep track of states for memory purposes in this implementation.

Thanks for your answer. I am gonna read the crate code to catch more details.