Why caret train function always return one .outcome when the model can have more than one dependent variable?

[paleomariomm]

This is something very tricky and is related with other questions I did in StackOverflow (1, 2, 3). The data to use is in 1.

Basically, I am comparing two neural network trainings:

1. One using directly the function neuralnet() from the neuralnet package.
2. One using the function train() from the caret package, and including method = "neuralnet".

The same formula in both models

In both cases, in the arguments to introduce the formula:

• f <- as.formula(paste("DC1 + DC2 + DC3 ~", paste(n[!n %in% c("DC1","DC2","DC3")], collapse = " + ")))

If we see its structure, you can see that there are three DEPENDENT VARIABLES (DC1, DC2, DC3)

str(f)
Class 'formula'  language DC1 + DC2 + DC3 ~ DC4 + DC5 + DC6 + DC7 + DC8 + DC9 + DC10
  ..- attr(*, ".Environment")=<environment: R_GlobalEnv> 

Using neuralnet() (neuralnet)

This is the code

BestModel <- neuralnet(formula = f, 
                 data = train_df,  # available in one of the links above
                 hidden = c(3,2,4) # random neurons per layer     
                 learningrate = 0.01, 
                 threshold = 0.01,   
                 stepmax = 50000       

)

Using train() (caret)

This is the code (all the parameters are available in this question):

model <- train(f, data = predict(pre_mdl_df, train_df), 
               method = "neuralnet", 
               tuneGrid = tune.grid.neuralnet,
               metric = "RMSE",
               stepmax = 100000,
               learningrate = 0.01,  
               threshold = 0.01,
               act.fct = softplus,
               trControl = caret::trainControl (
                 method = "repeatedcv", 
                 number = 2,               # Number of folds of the cv
                 repeats = 1,              # Number of cv repetitions
                 verboseIter = TRUE,
                 savePredictions = TRUE,
                 allowParallel = TRUE))

Here, we store as finalModel the final and best model obtained in the training:

finalModel <- model$finalModel

Comparison of the outputs from neuralnet() and train()

Different structures

• neuralnet(): str(BestModel) is a List of 14.
• train(): str(FinalModel) is a List of 19

Different responses:

As you can see, the response from neuralnet() is having chr [1:3] "DC1" "DC2" "DC3", whereas from train() has chr ".outcome". I guess this will impact the results later, because the prediction will return different quantity of results.

# neuralnet()
str(BestModel$response)
 num [1:26, 1:3] 4 5 6 8 11 11 11 6 8 5 ...
 - attr(*, "dimnames")=List of 2
  ..$ : chr [1:26] "165" "167" "168" "172" ...
  ..$ : chr [1:3] "DC1" "DC2" "DC3"

# train()
str(finalModel$response)
 num [1:26, 1] 0.83 0.509 1.199 1.353 1.55 ...
 - attr(*, "dimnames")=List of 2
  ..$ : chr [1:26] "X165" "X167" "X168" "X172" ...
  ..$ : chr ".outcome"

Different model lists

# neuralnet()
str(BestModel$model.list)
List of 2
 $ response : chr [1:3] "DC1" "DC2" "DC3"
 $ variables: chr [1:7] "DC4" "DC5" "DC6" "DC7" ...

# train()
str(finalModel$model.list)
List of 2
 $ response : chr ".outcome"
 $ variables: chr [1:7] "DC4" "DC5" "DC6" "DC7" ...

Different plots

• From neuralnet(): as expected, with three outputs representing the three dependent variables (DC1, DC2, DC3).

• From train(): only one output, .outcome.

Different prediction variables

I expect to predict the three variables simultaneously, as it happens with neuralnet() but not with train(), that returns only one.

# neuralnet()
head(predict(BestModel, train_df))
        [,1]     [,2]     [,3]
165 9.384303 10.38448 10.07678
167 9.384303 10.38448 10.07678
168 9.384303 10.38448 10.07678
172 9.384303 10.38448 10.07678
174 9.384303 10.38448 10.07678
176 9.384303 10.38448 10.07678

# train()
head(predict(finalModel, train_df))
         [,1]
165 0.8641611
167 1.3874627
168 0.8641579
172 1.1318320
174 0.8639903
176 1.7193986

Key questions

After comparing both models, which were supposed to behave similarly, several questions come to my mind:

• How can I make train() to work similarly as neuralnet()? This includes the plots and predictions.
• Is there any way to modify the code from train()?
• Is there a bug in train()?
• How can I change .outcome to have the three dependent variables?

Please, I need help!