Testing: Deeper Hidden Layers

[pmanion0]

Testing for a few different combinations of hidden layer size + depth.

Tasks:

• [x] Create test configuration for different hidden layer 1 + 2 sizes
• [x] Run the tests
• [x] Summarize the results

[dxl0632]

@theforager How do I set depth?

hiddendim: 10, 20

Looks like so

  public void setHiddenDimensions(String dimString) {
    String[] splitString = dimString.split(PARAMETER_DELIMITER);
    int[] dims = new int[splitString.length];
    for (int i=0; i < splitString.length; i++)
      dims[i] = Integer.valueOf(splitString[i]);
    setHiddenDimensions(dims);
  }

[pmanion0]

I think it will work that way, but try it without spaces to be safe (e.g. hiddendim: 10,20).

  private final String PARAMETER_DELIMITER = ",";
...
  public void setHiddenDimensions(String dimString) {
    String[] splitString = dimString.split(PARAMETER_DELIMITER);
    int[] dims = new int[splitString.length];
    for (int i=0; i < splitString.length; i++)
      dims[i] = Integer.valueOf(splitString[i]);
    setHiddenDimensions(dims);
  }

[dxl0632]

----optimal Run
  train:      ../data/train
  test:       ../data/dev
  wordvec:    ../data/wordVectors.txt
  vocab:      ../data/vocab.txt
  windowsize: 5
  wordvecdim: 50
  lowcaseall: true
  hiddendim:  50,50
  learnwordvec: true
  lambda:     0.001
  iters:      20
  trainevalfreq: 1
  learnrate:  0.03
----
  hiddendim: 50,150
----
  hiddendim: 50,300
----
  hiddendim: 100,50
----
  hiddendim: 100,150
----
  hiddendim: 100,300
----

[dxl0632]

A two-layer NN performs better than only single layer. But more layers doesn't mean better performance. This is only applicable this particular task and of course, I haven't tuned the parameter space exhaustively.