NN Model reloading Prediction Result Issue

[amnsms]

What's wrong?

I trained a Neural network model based on the training set then saved the model and reloaded the model into a new canvas with the same data used for training and the prediction node did not produce the same result used for training the NN model although the same data were fed into both.. I found others face the same issue of the loaded model not producing the same result as the trained model (i.e. issue #3521)

How can we reproduce the problem?

What's your environment?

• Operating system: win 10
• Orange version: 3.29
• How you installed Orange: 64 bit installer

[ajdapretnar]

I have tried to replicate this, but I can't (on heart disease data). I get the same predictions both time.

Could you perhaps replicate this with a data set that comes with Orange (in File or Datasets)? And please provide a screenshot of the workflow.

I see you are using Orange 3.29, but is it possible you have "Replicable training" in NN unchecked?

[amnsms]

Hi there, thank you for your responce. Attached is the workflow and replicable training is checked.

[markotoplak]

And what exactly are you doing then with the saved model? And how do you compare results?

[ajdapretnar]

Thanks! Perhaps just a question before we continue. It looks like you perhaps want to compare predictions from Test and Score (the ones in Data Table (3)) with Predictions from the saved model. If this is the case, it is entirely expected for the results to be different.

[amnsms]

after saving the model from 1st workflow,, I run 2nd workflow that use exactly same data used for training 1st wf and load the model and prediction is totally different from 1st wf test&score node

[ajdapretnar]

Yes, and this is entirely expected. Predictions were not made in the same way. The first is the result of k-fold cross-validation, while the second predictions (with the saved model) are done with a single model trained on the input data.

[amnsms]

so if a NN model training result shows 70% accuracy I expect it to produce a similar result on new data and if I used the same data used for training the model should even give closer results .. what shall I do then?

[ajdapretnar]

so if a NN model training result shows 70% accuracy I expect it to produce a similar result on new data

This is not true. You would need to test the model on a separate test data to estimate model's performance on unseen data.

As mentioned above, what you are seeing in Test and Score are results from cross-validation, meaning a different model is used to predict instances in each fold. In the saved model, a single model was trained on the input data and then used to predict it. So you have 5 models (if you are using 5-fold CV) vs. 1 model. Not the same thing.

[amnsms]

I assume orange will save the best parameters of the NN iteration that produces the highest score. So I did adjust the wf to include data sampler in both workflows (training/testing and prediction). I used a fixed portion of data (60%) for training and the remaining 40% for testing in the 1st workflow where the best NN settings should be saved in via the (save model) node. Then on the 2nd WF I also added data sampler node and used fixed remaining 20% of data for the prediction. I attached below both results. The model should produce high result on the 2nd wf as it already saw the data before, I mean if I train you to remember 5 words and then I show you 1st letter of them it will be much easier for you to remember those 5 words.. on the other hand (maybe my example is not perfect but this should be a good example of curve fitting)

[ajdapretnar]

I am not sure what you are trying to argue here.

I assume orange will save the best parameters of the NN iteration that produces the highest score. No, you just saved a model trained on the given data with the given parameters.

And you, your model likely has not seen the data before, because the model was trained on in-sample, while the predictions were made on out-of-sample.