arrayfire / arrayfire-ml

ArrayFire's Machine Learning Library.
BSD 3-Clause "New" or "Revised" License
102 stars 23 forks source link

Remove bias from Weights.hpp #25

Closed jramapuram closed 9 years ago

jramapuram commented 9 years ago

Weights are currently initializing bias' This is not always desirable, eg LSTM:

            /* Standard LSTM Implementation
             * Reference: http://deeplearning.net/tutorial/lstm.html
             * Refernece: http://www.cs.toronto.edu/~graves/phd.pdf */
            Weights mWi, mUi, mIDiffs;   // input gate weights + recurrence
            Weights mWf, mUf, mFDiffs;   // forget gate weights + recurrence
            Weights mWc, mUc, mCDiffs;   // memory cell weights + recurrence
            Weights mWo, mUo, mODiffs;   // output gate weights + recurrence

Here, each node needs only needs 1 bias, i.e. 1 bias for input gate (as opposed to the 2 we will now have for mWi & mUi) Either we should :

  1. refactor the above file into 2 files or
  2. we can provide a way to NOT initialize the bias in the CTOR

To me 1. makes more sense

pavanky commented 9 years ago

I think it is best to have just one Weights class. I see no reason for them to be combined. I think I just did what I did because it was easier during the prototype phase of LinearNode.

jramapuram commented 9 years ago

:+1:

pavanky commented 9 years ago

Fixed in master.