EVMS: Integrate core framework

[dlersch]

Dara Parser

Started implementing an image to numpy parser. All this parser does, is loading .png files and combining them into one .npy file. Attached is a screenshot of the unit-test. The unit-test just does an dimensional check, i.e. tests if the output dimensions of the numpy array math the user settings.

[dlersch]

Data Preperation

Added simple linear scaler which transforms a given numpy array X according to: A*X + B where the constants A,B are defined in the user config. The unit-test creates a numpy array, runs the transformation and reverses it. Dimensional checks ensure that the unit-test was successful. Attached is a screenshot of the unit-test

[dlersch]

Model

Started working on the module module by implementing a keras CNN Autoencoder. The model can be run with / without a keras early stopping callback. The corresponding unit-test trains the AE on a fraction of the mnist data which can be called via this helper class. Shown below are the plots that are produced by the unit-test. Please note: The model is not trained until perfect convergence. The idea of the unit-test is just to test the mechanics of the model, i.e. nothing crashes and all the functionality is available.

[dlersch]

Analysis

Implemented a residual analyzer analysis module. All this module does is to compute the difference between the prediction of a model and its input data:

residual = reduction_op[x - model(x),axis=reduction_axis]

where reduction_op is one of the following operations: mean (default), absolute value, squared value. The reduction_axis defines along which axis the reduction operation shall be performed. Since this project is predominantly dealing with images, the default is: reduction_axis = 3. Both, reduction_op and reduction_axis can be adjusted via a config file. The residual analyzer comes with its own unit-test where the MNIST data is folded with a gaussian distribution. The original and folded data (aka reconstructed data) are then fed into the residual analyzer which produces diagnostic plots, such as the two shown below:

[dlersch]

Entire Worflow / Driver

Started to implement the driver. Using @sgoldenCS graph module.

[dlersch]

Analysis: Data Reconstruction

Added a data reconstruction module which just takes a model m, input data x and collects the output:

y = m(x)

The data is handled via tensorflows dataset. This allows to handle large data sets, but also adds another level of flexibility to the workflow. The data reconstruction module allows to track the original data as well (in case the user decides to process only a small fraction of x). The data reconstruction analysis module was tested via its unit-test. A screenshot of said unit-test is attached.

The idea behind this module is that it can be combined with any other analysis module that requires a set of reconstructed data samples.

[dlersch]

Analysis: Learning Curve Visualizer

Added analysis module which visualizes learning curves. The input for this module is a dictionary, such as tensorflows history object. This modul enables the user to produces as many plots as needed and to visualize all metrics that are recorded duirng the training of a model. This analysis module comes with a dedicated unit-test, in which a dictionary with mock-data is passed to the analyzer. The mock data consists of four subsets that are separated into two groups which are then individually visualized.

Shown below are the outputs of the unit-test.

[dlersch]

Data Parser: MNIST Data Parser

Add MNIST data parsing module which does not require any inputs, but produces and output which is simply the MNIST digits. The idea behind this parser is, that it can be used to test and debug a workflow or just a specific module. Since this parser does not require any user inputs it can be used right away for testing or a small analysis. Furthermore will this module be helpful for quick demonstrations. This module has a dedicated unit-test which loads the MNIST data via said parser and visualizes it. Shown below is the example output of such a unit-test.

[dlersch]

Image Anomaly Detection Workflow

As mentioned last week, I am using the graph building tool written by @sgoldenCS to define a image anomaly detection driver. This driver uses all components that have been introduced in the previous posts. The driver consists of the following steps:

1. Use MNIST data parser to load MNIST digits
2. Scale MNIST data via the numpy linear scaler
3. Train a keras CNN AE on the scaled MNIST data
4. Use the learning curve visualizer to plot the training loss curves
5. Feed the scaled MNIST data and the trained AE into the data reconstruction module
6. Re-scale the reconstructed data back via the numpy linear scaler
7. Finally use the residual analyzer on the re-scaled data and plot residuals, i.e. the difference between the true and reconstructed MNIST digits.

Lines 14 - 23 nicely demonstrate how the above 7 steps define a sequential graph in the anomaly detection driver

Shown below are the results produced by the anomaly detection driver. Please note, that the AE was not tuned at all. I used best guess values to train the model.

[dlersch]

Hyper Parameter Optimization for Keras CNN AE

Implemented a hyper parameter tuner for the keras CNN AE. The tuner is based on the optuna framework. The key idea behind this implementation is, that the hyper parameter tuner inherits from the model core and is therefore its own model module. Consequently, this model is also registered as a model. The registration for the keras CNN AE hyper parameter tuner can be found here. This implementation comes with its own default configuration and unit-test. The unit-test runs a very short and basic analysis. Shown below are a few results reported by the unit-test. The keras CNN AE was tuned for 10 trials on the MNIST data and the final / best model was trained for 20 epochs.