mem_cell_ct =100 but only 4 LSTM nodes

[ichraibi]

Hi, I was wondering how did you choose the number of memory cells? And why and how did you choose 100 as a value for your parameter?

And finally, just to be clear : Do you create in the code, 4 LSTM nodes, because of the 4 "input sequence- target value" pairs?

And what is the difference between a memory cell and a LSTM node? I've quickly read the paper that you recommand in your blog, but it's still not clear for me.

Thanks in advance for your answers. Ick

[ScottMackay2]

As far as I can see, he choose for 100 hidden cells because that is the double of the randomly chosen 50 input cells.

Choosing the amount of hidden cells could be a bit of a guessing game. Trying different numbers to see what works best. This is the case for most neural networks (as far as I know of, all of them)

There are tricks that makes the guessing simpler. Like, making the hidden layer use twice as much cells as input cells (as in this example). Or choosing the average of the input + output cells. I for example tested it with 26 ((50inputs + 1output) / 2) hidden cells. And that resulted in about the same loss.

But there are also networks that try to find the amount of hidden cells for them self. More explanation can be found in this thread: http://stackoverflow.com/questions/3345079/estimating-the-number-of-neurons-and-number-of-layers-of-an-artificial-neural-ne

About LSTM nodes. They seem to me that their only use is to remember the input at a certain time step and couple that with the states of all cells of the LstmState object. This way the back propagation could work. (you need the input values times the difference to find the error of the weights). Summary in the program: LstmNode is used to couple self.state (all the values in all the cells) with self.xc (the input).

[xiaohu2015]

In fact, the project only uses one memory cell, which has big difference from LSTM node. The memory cell in this work really means the hidden size of the cell.

[zackchase]

This is not true. Likely you've been confused by Tensorflow's inaccurate notation. A single LSTM node is called a cell (in the literature).

On Feb 5, 2017 7:51 PM, "xiaohu" notifications@github.com wrote:

In fact, the project only uses one memory cell, which has big difference from LSTM node. The memory cell in this work really means the hidden size of the cell.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nicodjimenez/lstm/issues/12#issuecomment-277581012, or mute the thread https://github.com/notifications/unsubscribe-auth/ACR4zqjo6QsQNHbcBDpP-frnDAV_Tk4Dks5rZpjDgaJpZM4LnI_q .

[xiaohu2015]

I think the notation in the literature is not consistent. I refer to https://arxiv.org/abs/1506.00019

[jsbhat]

@zackchase As far I understood, the terminology can be confusing. However, what @ScottMackay2 writes makes sense to me from the code and implementation point of view of unrolling the LSTM network.

In lstm.py, LSTM nodes refer to the repeating (same weight parameters, varying cell states) hidden layer of memory cells in the unrolled network. In example_0, the number of nodes is set programmatically to the maximum number (n=4) of time steps of the random 50 dimensional input sequence.