`neuralweights` extracts coefficients in wrong order from `nnet` package

[TommyJones]

Hi. It seems to me that using neuralweights to extract weights from a network from the nnet package results in the weights being placed in the wrong order. The result is that if you use these weights for manual prediction, the predictions will be wrong. (Note that I believe this issue is exacerbated by poor documentation on nnet's part.)

I've included a reproducible example below.

library(nnet)

library(NeuralNetTools) 

# use the iris dataset. It's small and has 3 outcomes
dat <- iris

# train the model
f <- multinom(Species ~ ., data = dat)

plotnet(f)

# make a prediction with predict() as a baseline
p_f <- predict(f, dat, type = "prob")

# extract weights with neuralweights()
w_nnt <- neuralweights(f)

# do a manual prediction with w_nnt and softmax model
p_f_nnt <- cbind(1, 1, as.matrix(dat[, -5])) %*% t(do.call(rbind, w_nnt[[2]]))

p_f_nnt <- exp(p_f_nnt) / rowSums(p_f_nnt)

# we get very different predictions
# compare with the correlation coefficient
for (j in 1:ncol(p_f)) {
  print(cor(p_f[, j], p_f_nnt[, j]))
}

# extract the weights using coef
# coefficients for the first outcome are zero for all variables as a baseline
# not sure why they did it this way. This is the first time I've ever seen it
# done this way. 
w_f <- rbind(0, coef(f)) 

rownames(w_f)[1] <- "setosa" 

# get prediction and
# apply the softmax function to get probabilities
p_f_manual <- cbind(1, as.matrix(dat[, -5])) %*% t(w_f)

p_f_manual <- exp(p_f_manual) / rowSums(exp(p_f_manual))

# we get the same predictions
# compare with the correlation coefficient
for (j in 1:ncol(p_f))
  print(cor(p_f[, j], p_f_manual[, j]))

[fawda123]

@TommyJones Thanks for the reprex. The documentation for the nnet package is a bit sparse.

Looks like you are using a multinomial model, via nnet. I haven't tested the package with these models at all, so the order of the weights might be incorrect. Have you checked the order without using a classification model? Could just be that I need to add a disclaimer that the functions don't work for multinomial models.

[TommyJones]

I have not tried it with a non-multinomial classification model. But thank you for the tool. I was quite stuck and this package got me unstuck, even if the weights weren't in the right place.

On Mon, Apr 6, 2020 at 8:17 AM Marcus W Beck notifications@github.com wrote:

@TommyJones https://github.com/TommyJones Thanks for the reprex. The documentation for the nnet package is a bit sparse.

Looks like you are using a multinomial model, via nnet. I haven't tested the package with these models at all, so the order of the weights might be incorrect. Have you checked the order without using a classification model? Could just be that I need to add a disclaimer that the functions don't work for multinomial models.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fawda123/NeuralNetTools/issues/22#issuecomment-609758233, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGCQAXI3BUHIUQ327OSTK3RLHB4DANCNFSM4LPKRYAQ .

[fawda123]

If it helps, you can see how the weights are extracted here: https://github.com/fawda123/NeuralNetTools/blob/eb2f60ab7c16bb57a2a7bf006279f4a4bf6c4e4b/R/NeuralNetTools_utils.R#L104

It's super hacky, but it gives the correct order needed for the rest of the functions in the package. Model predictions are another question...