Systematic testing capability

[amartinhuertas]

Let me open a different issue to have specialized discussions in regards to this.

Let me start by copying and pasting @GiteonCaulfied comment in the other issue.

Hi @amartinhuertas @rhyshawkins

I've implemented a naive systematic testing method, including a new notebook Systematic_testing.ipynb and a text file ModelList.txt that contains a list of different hyperparameters settings for the specific result_20k_zero dataset (including hidden layers neurons, activation function, Loss function type, epoch, batch size and learning rate)

By running the Systematic_testing.ipynb, it will read each line of the ModelList.txt file to build a NN model that aligns with the settings specified in that line. After that, the plot of training loss and validation loss during training, also the plot of the Best/Worst case input and output for training set and testing set will be generated when the code is running.

One thing about this naive systematic testing method is that these plots for different models are not put together in a complete grid structure after each model has been trained and tested. It also can't be selected and viewed with a drop down box in an interactive way like the one plot in the given Julia notebook. Therefore, to compare the performance of each model, one may need to keep scrolling down and scrolling up and that can be kind of inconvenient. I'll try to think of some ways to make it better if possible.

The updated version of code will be uploaded shortly after this comment as usual.

[amartinhuertas]

Some notes:

• When a given combination of hyperparameters are tested, do you store in a file the resulting trained machine learning architecture using a name that uniquely defines the experiment?
• Assuming that the previous question is affirmative, do you avoid running again a given combination of hyperparameters when the corresponding file is already available?
• I would also commit&push the files associated to the trained NN archictectures, possibly tagging them with the git repo commit id.

[amartinhuertas]

Therefore, to compare the performance of each model, one may need to keep scrolling down and scrolling up and that can be kind of inconvenient. I'll try to think of some ways to make it better if possible.

Another alternative is to open the same notebook in different browser tabs, and then visualize simultaneously different models in different tabs split vertically. Does that make sense?

[GiteonCaulfied]

Hi @amartinhuertas

I've modified my implementation for Systematic testing capability. The main files now includes systematic_testing.ipynb,ModelList.txt and st1D_template.ipynb. By running systematic_testing.ipynb, it will read through each line of ModelList.txt and generate a new Jupyter Notebook for each set of parameters by changing the values of parameters tagged in the st1D_template.ipynb file (This newly generated notebook, or model, will be executed automatically so that one can refer to its result later without running it manually).

In this case, different models will be trained and tested in different Jupyter Notebook and we can compare the results simply by opening them and then switching tabs.

This procure makes use of an external package called papermill, which you need to install before running the systematic_testing.ipynb file (not needed for running the newly generated models, though), here's a link to more details about this package:

https://papermill.readthedocs.io/en/latest/

Also, I've added some other parameters in ModelList.txt, including input/output data path, baseline of loss deciding whether a prediction is accurate, scaler applied to the output data and also the name of the model. I haven't come up with a way to make the name of model being both simple and meaningful, so I'll just name it manually in this case. (For example, st1D-invGeoid.ipynb is generated from the template file st1D_template.ipynb, and use small_8_1k-inv.npy as input and 'small_8_1k-geoid.npy' as output to train the model, I also want to add the hidden layers structure to the name of the model but I will just leave it here since I've only got one model here)

The updated code will pushed shortly after this comment, I will also write a short comment about the performance of this st1D-invGeoid.ipynb model in the first issue.

[amartinhuertas]

I was thinking on something slightly different.

Namely, the trained NN architecture is stored in a light-weight file, in a folder tagged with the commit id corresponding to the version of the code that was used to generate the NN. I guess that PyTorch has tools to export a NN to a file, right?

Then, you only need a single notebook to visualize the results. This visualization notebook contains a cell in which you specify the path to the NN file mentioned in the previous paragraph. Then, you can open the same notebook on several tab browsers, and modify the value of this cell as per required.

[amartinhuertas]

Namely, the trained NN architecture is stored in a light-weight file, in a folder tagged with the commit id corresponding to the version of the code that was used to generate the NN. I guess that PyTorch has tools to export a NN to a file, right?

Of course, apart from the NN parameters itself, you may also need to store any dianostics information about the training process itself, e.g., in a bson file.

[amartinhuertas]

Associating all diagnostic/NN files to commit IDs will be key to handle tracking of the process in an educated way/extensible way.

For example, you may need to add extra parameters to ModelList.txt in the future (e.g., regularization term). The old model files, i.e., the ones that you trained with an old version of ModelList.txt, are no longer valid for the new ModelList.txt, but that's ok as we are keeping a record on the association among commit IDs and trained models.

[GiteonCaulfied]

Hi @amartinhuertas

Namely, the trained NN architecture is stored in a light-weight file, in a folder tagged with the commit id corresponding to the version of the code that was used to generate the NN. I guess that PyTorch has tools to export a NN to a file, right?

Then, you only need a single notebook to visualize the results. This visualization notebook contains a cell in which you specify the path to the NN file mentioned in the previous paragraph. Then, you can open the same notebook on several tab browsers, and modify the value of this cell as per required.

I've create another version of the systematic testing method that aligns with this comment. Two files are related in this process, systematic_training.ipynb and model_visualisation.ipynb, where systematic_training.ipynb will fetch all the hyperparameters in the ModelLists.txt line by line and train these models. The trained NN architecture along with a text file contains the training loss and validation loss during training will be stored in the 1D_results directory. The name of these files will be a long string specifies each of the hyperparameters I used to train the model.

For testing and visualisation, simply open systematic_training.ipynb and replace the value of the model_path variable in the second cell to a model you would like to test with, then run the entire notebook, the testing result along with the plots of training loss and validation loss will be shown cell by cell.

For the initial systematic testing method I've created, I want to keep it a bit longer since this one is easier to check result. To avoid confusion, I changed the name of the original file Systematic_testing.ipynb to notebook_systematic_testing.ipynb. I also put the all the output notebook and the template notebook to a folder called 1D_result_notebook.

The new systematic testing method can't check if a training is done before yet, it also can't associate a group of result model to a certain commit ID (for this one I am actually thinking about doing it manually. For example, put the result file in a unique folder after I commit and push). I'll implement these things later.

I've also tested the new 1k_zero dataset provided on the Google Drive, I'll talk about that on the first issue.

The updated code will pushed shortly after this comment.

[GiteonCaulfied]

Hi @amartinhuertas

I've moved the training models in the last commit in a folder named after the commit ID. For now, each commit will include some newly training models in the path 1D_results and move the models from the last commit in a unique folder so that their path now become 1D_results/last_commit_ID

I've also implemented a simple way to check if a certain training is done before in the file systematic_training.ipynb, as well as a new method to display accuracy under different threshold for loss value.

The updated files have been pushed to the repo.