Pretrained autoencoder-deeplearning POJO model vs. in-h2o probability mismatch

[exalate-issue-sync[bot]]

https://groups.google.com/forum/#!topic/h2ostream/sLtl4X2DfCI

[exalate-issue-sync[bot]]

Arno Candel commented: Unable to repro: see https://github.com/h2oai/h2o-3/blob/master/h2o-algos/src/test/java/hex/deeplearning/DeepLearningTest.java#L1596-L1596

[exalate-issue-sync[bot]]

Lauren DiPerna commented: if you download the attached files you can then run: {code} $ javac -cp h2o-genmodel.jar -J-Xmx2g -J-XX:MaxPermSize=128m DeepLearning_model_R_1469480193092_2.java main.java $ java -cp .:h2o-genmodel.jar main {code}

main.java corresponds to row 19 in the saved_predictions.csv file

+The attached files:+ pojo_testing_script.R: generates two binary files (the first saves a model with autoencoder tuning, the second saves a model with stacked autoencoder-deeplearning), and then downloads a pojo of the later [paths to the location where these files are saved needs to be updated to run the script].

saved_predictions.csv: exported results of h2o.predict(model, test_data), and compares a few row predictions from h2o with the pojo results

cbb_test.RDS and cdd_train.RDS: are datasets needed to run the pojo_testing_script.R if you want to regenerate the binary files and pojo.

[exalate-issue-sync[bot]]

Arno Candel commented: Still can't repro, JUnit with identical parameters passes fine.

The model has 352 inputs, but only 162 are filled in main.java, so that's clearly wrong.

arno@lappy:~/Downloads$ grep row.put main.java | wc -l 162

[exalate-issue-sync[bot]]

Arno Candel commented: Here's the code for the passing JUnit: {code} @Test public void test_PUBDEV3168() { Frame train = null; Frame test = null; DeepLearningModel dl1 = null; DeepLearningModel dl2 = null; DeepLearningModel ae = null;

try {

// > setwd("/Users/arno/Downloads/") // > train <- readRDS("cbb_train.RDS") // > test <- readRDS("cbb_test.RDS") // > write.csv(train,"cbb_train.csv") // > write.csv(test,"cbb_test.csv") train = parse_test_file("/users/arno/Downloads/cbb_train.csv"); test = parse_test_file("/users/arno/Downloads/cbb_test.csv");

  // train unsupervised AE
  Key<DeepLearningModel> key = Key.make("ae_model");
  {
    DeepLearningParameters parms = new DeepLearningParameters();
    parms._ignored_columns = new String[]{"class"};
    parms._activation = DeepLearningParameters.Activation.TanhWithDropout;
    parms._autoencoder = true;
    parms._epochs = 20;
    parms._hidden = new int[]{380,260,180,120,60,8};
    parms._input_dropout_ratio = 0.05;
    parms._hidden_dropout_ratios = new double[]{0.30,0.25,0.20,0.10,0.01,0};
    parms._max_w2 = 1e5f;
    parms._train = train._key;
    parms._seed = 1234;
    ae = new DeepLearning(parms, key).trainModel().get();
    // test POJO
    Frame res = ae.score(test);
    assertTrue(ae.testJavaScoring(test, res, 1e-5));
    res.remove();
  }

  // train supervised DL model
  {
    DeepLearningParameters parms = new DeepLearningParameters();
    parms._pretrained_autoencoder = key;
    parms._response_column = "class";
    parms._activation = DeepLearningParameters.Activation.TanhWithDropout;
    parms._epochs = 20;
    parms._hidden = new int[]{380,260,180,120,60,8};
    parms._input_dropout_ratio = 0.01;
    parms._hidden_dropout_ratios = new double[]{0.30,0.25,0.20,0.10,0.05,0};
    parms._train = train._key;
    parms._seed = 1234;
    parms._adaptive_rate = false;
    parms._initial_weight_distribution = DeepLearningParameters.InitialWeightDistribution.Uniform; //ignored
    parms._rate = 0.005;
    parms._rate_annealing = 1e-3;
    parms._momentum_start = 0.1;
    parms._momentum_ramp = 2; //too small - 2 samples
    parms._momentum_stable = 1e-5;

    dl1 = new DeepLearning(parms).trainModel().get();
    // test POJO
    Frame res = dl1.score(test);
    assertTrue(dl1.testJavaScoring(test, res, 1e-5));
    res.remove();
  }

} finally {
  if (train != null) train.delete();
  if (test != null) test.delete();
  if (ae != null) ae.delete();
  if (dl1 != null) dl1.delete();
  if (dl2 != null) dl2.delete();
}

} {code}

[exalate-issue-sync[bot]]

Arno Candel commented: A few comments:

This is not a stacked AE, but a pretrained AE. The hidden layers go all the way down to 8 neurons, but then it jumps back up to 352, which is not ideal. Initial distribution is meaningless for the second model that starts from the pretrained model. No need for fancy name extraction, just set model_id="ae" in the first model, then use pretrained_autoencoder="ae" in the second.

[exalate-issue-sync[bot]]

Arno Candel commented: Unable to repro: see https://github.com/h2oai/h2o-3/blob/master/h2o-algos/src/test/java/hex/deeplearning/DeepLearningTest.java#L1596-L1596

[exalate-issue-sync[bot]]

Lauren DiPerna commented: if you download the attached files you can then run: {code} $ javac -cp h2o-genmodel.jar -J-Xmx2g -J-XX:MaxPermSize=128m DeepLearning_model_R_1469480193092_2.java main.java $ java -cp .:h2o-genmodel.jar main {code}

main.java corresponds to row 19 in the saved_predictions.csv file

+The attached files:+ pojo_testing_script.R: generates two binary files (the first saves a model with autoencoder tuning, the second saves a model with stacked autoencoder-deeplearning), and then downloads a pojo of the later [paths to the location where these files are saved needs to be updated to run the script].

saved_predictions.csv: exported results of h2o.predict(model, test_data), and compares a few row predictions from h2o with the pojo results

cbb_test.RDS and cdd_train.RDS: are datasets needed to run the pojo_testing_script.R if you want to regenerate the binary files and pojo.

[exalate-issue-sync[bot]]

Arno Candel commented: Still can't repro, JUnit with identical parameters passes fine.

The model has 352 inputs, but only 162 are filled in main.java, so that's clearly wrong.

arno@lappy:~/Downloads$ grep row.put main.java | wc -l 162

[exalate-issue-sync[bot]]

Arno Candel commented: Here's the code for the passing JUnit: {code} @Test public void test_PUBDEV3168() { Frame train = null; Frame test = null; DeepLearningModel dl1 = null; DeepLearningModel dl2 = null; DeepLearningModel ae = null;

try {

// > setwd("/Users/arno/Downloads/") // > train <- readRDS("cbb_train.RDS") // > test <- readRDS("cbb_test.RDS") // > write.csv(train,"cbb_train.csv") // > write.csv(test,"cbb_test.csv") train = parse_test_file("/users/arno/Downloads/cbb_train.csv"); test = parse_test_file("/users/arno/Downloads/cbb_test.csv");

  // train unsupervised AE
  Key<DeepLearningModel> key = Key.make("ae_model");
  {
    DeepLearningParameters parms = new DeepLearningParameters();
    parms._ignored_columns = new String[]{"class"};
    parms._activation = DeepLearningParameters.Activation.TanhWithDropout;
    parms._autoencoder = true;
    parms._epochs = 20;
    parms._hidden = new int[]{380,260,180,120,60,8};
    parms._input_dropout_ratio = 0.05;
    parms._hidden_dropout_ratios = new double[]{0.30,0.25,0.20,0.10,0.01,0};
    parms._max_w2 = 1e5f;
    parms._train = train._key;
    parms._seed = 1234;
    ae = new DeepLearning(parms, key).trainModel().get();
    // test POJO
    Frame res = ae.score(test);
    assertTrue(ae.testJavaScoring(test, res, 1e-5));
    res.remove();
  }

  // train supervised DL model
  {
    DeepLearningParameters parms = new DeepLearningParameters();
    parms._pretrained_autoencoder = key;
    parms._response_column = "class";
    parms._activation = DeepLearningParameters.Activation.TanhWithDropout;
    parms._epochs = 20;
    parms._hidden = new int[]{380,260,180,120,60,8};
    parms._input_dropout_ratio = 0.01;
    parms._hidden_dropout_ratios = new double[]{0.30,0.25,0.20,0.10,0.05,0};
    parms._train = train._key;
    parms._seed = 1234;
    parms._adaptive_rate = false;
    parms._initial_weight_distribution = DeepLearningParameters.InitialWeightDistribution.Uniform; //ignored
    parms._rate = 0.005;
    parms._rate_annealing = 1e-3;
    parms._momentum_start = 0.1;
    parms._momentum_ramp = 2; //too small - 2 samples
    parms._momentum_stable = 1e-5;

    dl1 = new DeepLearning(parms).trainModel().get();
    // test POJO
    Frame res = dl1.score(test);
    assertTrue(dl1.testJavaScoring(test, res, 1e-5));
    res.remove();
  }

} finally {
  if (train != null) train.delete();
  if (test != null) test.delete();
  if (ae != null) ae.delete();
  if (dl1 != null) dl1.delete();
  if (dl2 != null) dl2.delete();
}

} {code}

[exalate-issue-sync[bot]]

Arno Candel commented: A few comments:

This is not a stacked AE, but a pretrained AE. The hidden layers go all the way down to 8 neurons, but then it jumps back up to 352, which is not ideal. Initial distribution is meaningless for the second model that starts from the pretrained model. No need for fancy name extraction, just set model_id="ae" in the first model, then use pretrained_autoencoder="ae" in the second.

[hasithjp]

JIRA Issue Migration Info

Jira Issue: PUBDEV-3168 Assignee: Arno Candel Reporter: Arno Candel State: Resolved Fix Version: N/A Attachments: Available (Count: 10) Development PRs: N/A

Attachments From Jira

Attachment Name: cbb_test.RDS Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/cbb_test.RDS

Attachment Name: cbb_train.RDS Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/cbb_train.RDS

Attachment Name: DeepLearning_model_R_1469480193092_1 Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/DeepLearning_model_R_1469480193092_1

Attachment Name: DeepLearning_model_R_1469480193092_2 Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/DeepLearning_model_R_1469480193092_2

Attachment Name: DeepLearning_model_R_1469480193092_2.java Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/DeepLearning_model_R_1469480193092_2.java

Attachment Name: h2o-genmodel.jar Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/h2o-genmodel.jar

Attachment Name: main.java Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/main.java

Attachment Name: pojo_testing_script.R Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/pojo_testing_script.R

Attachment Name: saved_predictions.csv Attached By: Lauren DiPerna File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/saved_predictions.csv

Attachment Name: Screen Shot 2016-07-27 at 12.34.38 PM.png Attached By: Arno Candel File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-3168/Screen Shot 2016-07-27 at 12.34.38 PM.png