Error getting variable importance with nnet object with skip-layer connections

[hongooi73]

I'm getting an eror trying to run garson() with an nnet object when skip-layer connections are present (ie, skip=TRUE). Here's a minimal example:

library(nnet)
nn2 <- nnet(Sepal.Length ~ Sepal.Width + Petal.Length + Petal.Width, data=iris, size=3,
    linout=TRUE, skip=TRUE)
garson(nn2, "Sepal.Length", bar_plot=FALSE)

Error in data.frame(wts, row_nms) : 
  arguments imply differing number of rows: 19, 16

Note that this isn't technically with the NeuralNetTools package loaded, as I'm still using R 3.1.1 (technically, Revolution R 7.3, but that uses R 3.1.1 under the hood) and the package lists 3.1.2 as the minimum version supported. Instead I extracted the source files from the archive and manually sourced them into my R session.

Is there something about the 3.1.1 -> 3.1.2 transition that could be causing this problem? I was actually using your code in a project in November and everything worked fine (although that wasn't with skip-layer connections present, so I didn't experience this bug).

Great work btw, this fills in a major hole for neural network functionality.

[fawda123]

Thanks for catching this issue. My guess is it's not related to the R version - probably just my faulty coding. I'll get to this in the next day or two.

[hongooi73]

Thanks! If you don't mind, could you also remove (or reduce) the R version limitation? Would make it much easier for me to use your code.

On Thu, Mar 5, 2015 at 10:45 PM, Marcus W Beck notifications@github.com wrote:

Thanks for catching this issue. My guess is it's not related to the R version - probably just my faulty coding. I'll get to this in the next day or two.

— Reply to this email directly or view it on GitHub https://github.com/fawda123/NeuralNetTools/issues/3#issuecomment-77375404 .

[fawda123]

No problem, the version was changed though technically it was built on 3.1.2.

I had a look at the issue and the problem was in the neuralweights function that extracted the weights from the model object. The additional weights that were provided with the skip layer messed up the output.

I'm not sure how to handle these additional weights since they are technically parameters that influence the values of the others weights (I-H, H-O) during model training. I took the easy way out and simply removed them when plotting or using garson's algorithm. The functions technically work but I have two concerns about this.

1) The shading of weights in the plot are relative to each other. The shading is not really incorrect if you consider only the weights that are shown in the plot but this may be misleading since the skip-layer weights also have to be considered for their relative magnitudes. I also couldn't think of an aesthetically nice way to show a skip-layer in the plot... so I took the easy way out and simply ignored them.

2) For similar reasons, Garson's algorithm relies on the relative magnitudes of all the weights. Again, inclusion of skip-layers will alter the values of all weight vectors during training and they should be considered when calculating variable importance. Formulaically, I have no idea how the skip-layer can be included in the algorithm. I don't think this is ever mentioned in the original paper that describes the method (though I will check when I get access on my other computer, may also email Garson)... so I took the easy way out and ignored the skip-layer connections.

I don't think that the approach is completely legitimate but I can't think of an easy fix. I've included a warning that will be printed in the console if a skip layer is used with the nnet or train functions which indicates that results may be inaccurate. Note that the lekprofile function shouldn't be affected since it only uses the predict method for the neural network, i.e., all weights are accounted for.

I don't think my quick fix is ideal since comparing variable importance from models with and without skip layers produces inconsistent results depending on the function used to create the network. The following code compares neural networks w/ and w/o skip layers using nnet, then using train. You'll see that the variable importance values are very different using nnet w/ and w/o skips but they are similar using train. I would of course have more faith in the variable importance values for the models that are fit without skip layers since they follow the Garson method exactly. The importance values for the models that include skip-layers are probably unreliable since the skip-layers are not accounted for in the calculation.

library(NeuralNetTools)

data(neuraldat)

library(nnet)
set.seed(1234)
nn1 <- nnet(Y1 ~ X1 + X2 + X3, method = 'nnet', data = neuraldat, linout = TRUE)
set.seed(1234)
nn2 <- nnet(Y1 ~ X1 + X2 + X3, method = 'nnet', data = neuraldat, linout = TRUE, skip = T)
garson(nn1, 'Y1', bar_plot = FALSE)
garson(nn2, 'Y1', bar_plot = FALSE)

library(caret)
set.seed(1234)
mod1 <- train(Y1 ~ X1 + X2 + X3, method = 'nnet', data = neuraldat, linout = TRUE)
set.seed(1234)
mod2 <- train(Y1 ~ X1 + X2 + X3, method = 'nnet', data = neuraldat, linout = TRUE, skip = T)
garson(mod1, 'Y1', bar_plot = FALSE)
garson(mod2, 'Y1', bar_plot = ALSE)

Anyway, sorry I couldn't find a better solution. At least this way the functions still work with skip layers, although probably not as they should.

[hongooi73]

Thanks for the reply. I haven't read the paper by Garson, but from the code, it looks like it's taking linear combinations of the coefficients at each neuron and summing them up for each variable. Couldn't you just add the values of the skip-layer coefficients to the values derived from the hidden layer stack?

This would also make the result consistent with the limit where the number of hidden units is zero, ie the NN turns into a linear model. Then the weights would simply be the linear regression coefficients.

[fawda123]

I'm keeping this issue open for now because I want to implement the changes we discussed over email. Stay tuned.

[fawda123]

Updated package is now on CRAN - includes an 'olden' function that can handle skip layers.