predicting memory requirements?

[jackbackes]

I'm fairly new to implementing neural networks and am using this library for the first time. I'm doing text processing and so far I'm up to 5.6 GB of RAM required on a simple bigram.

The way I'm structuring the training data is like this: { input: {[inputWord]: 1}, output: {[outputWord]: 1 } }

so if it were "cat sat" the training data would look like: { input: {cat: 1}, output: {sat: 1} }

I have two questions:

1. what's a good way to predict memory requirements? Is there a back-of-napkin big-O?
2. is there a better way to structure the data in this particular case?

Thanks!

[robertleeplummerjr]

Would you be in a position to share your code? Likely there is a memory leak that needs addressed.

[Macil]

Isn't that code going to make a new input neuron and output neuron for every single distinct word used? That might make a ton more neurons than expected if given a lot of words.

[robertleeplummerjr]

Just curious: Does your error rate drop fairly quick?

[robertleeplummerjr]

Also: How many words are we talking about training with?

[robertleeplummerjr]

To answer @AgentME, I believe you are totally correct. I honestly don't know if a feed forward neural net (currently the main thrust of brain.js, but with others potentially on the way, like recurrent neural nets: https://github.com/harthur-org/brain.js/tree/recurrent/lib/recurrent) is the best means of classification like you are currently doing. It probably needs backpropagation, so convolutional networks (https://en.wikipedia.org/wiki/Convolutional_neural_network) come to mind as a proposed solution: http://cs.stanford.edu/people/karpathy/convnetjs/ for the time being.

I wish brain.js had said type of network, and eventually I plan on it having it, if the community agrees it'd be ideal.

[robertleeplummerjr]

The other option would be to try out the recurrent neural network, but it isn't even alpha yet.

[robertleeplummerjr]

After thinking about this more, you may be able to do something like this (note: pseudo code):

var alphabet = {};
' abcdefghijklmnopqrstuvwxyz'
  .split()
  .forEach(function(letter, index) {
    alphabet[letter] = index;
  });

function wordToIndexes(word) {
  word = word.toLowerCase();
  var result = [];
  for (var i = 0; i < word.length; i++) {
    result.push(alphabet[word[i]]);
  }
  return result;
}

{
  input: wordToIndexes(inputWord),
  output: wordToIndexes(outputWord)
}

I don't think it'd be ideal, but the lightweightness of the feedforward neural net may give you something you desire.

[jackbackes]

@robertleeplummerjr: I ended up doing something similar to what you did above. Splitting it into letters kept me from having to have thousands of inputs, which is where the memory leak was.

Closing as solved.

[robertleeplummerjr]