problems with custom fitness function

[mytarmail]

Hey! thanks for the cool package! I want to apply my fitness functions, I decided to start with a simple example with correlation but the network is not trained What am I doing wrong? thanks.

data(iris)
xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species)))
ymat <- iris[,1]

f <- "cor(yhat,y)" 
amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
                                hpar = list(modexec = 'trainwpso',
                                            numiterations = 30,
                                            psopartpopsize = 50,
                                            costcustformul = f))

(cost: custom)
cost epoch10: 1.46448412698413 (cv cost: 1.46448412698413) (LR:  0 ) 
cost epoch20: 1.46448412698413 (cv cost: 1.46448412698413) (LR:  0 ) 
  [test trgainunder OK] [test cvgainunder OK]
[2 / 2 cond. OK to exit]
   dim X: [4,135]
   dim W1: [10,4] (min|max: -0.707813207710181, 1.22940301439856)
   dim bB1: [10,1] (min|max: -0.0172758234121704, 0.0251791206132498)
   dim W2: [1,10] (min|max: -0.00400016407025268, 0.017710438251683)
   dim bB2: [1,1] (min|max: -0.00469585680774184, -0.00469585680774184)
   dim Y: [1,135]

[aboulaboul]

Thanks for the comment. I only see OKs in the message you sent, what is the problem, did you try the predict function ?

[mytarmail]

Thank you for responding! I restarted the session and now I'm getting an error

library(automl)
data(iris)
xmat <- as.matrix(cbind(iris[,2:4], as.numeric(iris$Species)))
ymat <- iris[,1]
f <- "cor(yhat,y)" 
amlmodel <- automl_train_manual(Xref = xmat, Yref = ymat,
                                hpar = list(modexec = 'trainwpso',
                                            numiterations = 30,
                                            psopartpopsize = 50,
                                            costcustformul = f))

..

(cost: custom)
Error in sbamlmoddlcost(mydl, y = mylspsoparam$angdl$Y, yhat = Yhat, costtype = mydl$hpar$costtype,  : 
  object 'J' not found

Can you show me a working example on my data, and also explain the essence of the variables that you will use

[aboulaboul]

and what if you replace f <- "cor(yhat,y)" by f <- "J=cor(yhat,y)"

[mytarmail]

f <- "J=cor(yhat,y)" ..

(cost: custom)
Error in if (mylspsoactivpart$Cost < mylspsoactivpart$Best$Cost) { : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In if (mylspsoactivpart$Cost < mylspsoactivpart$Best$Cost) { :
  the condition has length > 1 and only the first element will be used

[aboulaboul]

It looks like cor produces NAs and/or more than one value, so with this, I replace NAs by a big value 1e10 and I limit result to a num with J[1] : f <- "J=cor(yhat,y)" f <- c(f, "J=ifelse(is.na(J),1e10, J)") f <- c(f, "J=J[1]") f <- paste(f, collapse = ';')

[mytarmail]

Thanks for the help, it works without errors, but the network does not learn anything anyway

f <- "J=cor(yhat,y)"
f <- c(f, "J=ifelse(is.na(J),1e10, J)")
f <- c(f, "J=J[1]")
f <- paste(f, collapse = ';')

..

(cost: custom)
cost epoch10: 1e+10 (cv cost: 1e+10) (LR:  0 ) 
cost epoch20: 1e+10 (cv cost: 1e+10) (LR:  0 ) 
  [test trgainunder OK] [test cvgainunder OK]
[2 / 2 cond. OK to exit]
   dim X: [4,135]
   dim W1: [10,4] (min|max: -0.707813207710181, 1.22940301439856)
   dim bB1: [10,1] (min|max: -0.0172758234121704, 0.0251791206132498)
   dim W2: [1,10] (min|max: -0.00400016407025268, 0.017710438251683)
   dim bB2: [1,1] (min|max: -0.00469585680774184, -0.00469585680774184)
   dim Y: [1,135]

cost epoch10: 1e+10 (cv cost: 1e+10) (LR: 0 ) cost epoch20: 1e+10 (cv cost: 1e+10) (LR: 0 )

Solution. The reason for the problems is that, for some reason that is not clear to me, the variables yhat and y are not vectors, although it seems intuitively that they should

I just made them vectors and the network started working as it should. f <- "J=cor( c(yhat) , c(y) )" ..

(cost: custom)
cost epoch10: -0.934871659246764 (cv cost: -0.879963013239127) (LR:  0 ) 
cost epoch20: -0.940606056654142 (cv cost: -0.885602684676496) (LR:  0 ) 
cost epoch30: -0.940778691889731 (cv cost: -0.874268481977938) (LR:  0 ) 
cost epoch40: -0.941234964839193 (cv cost: -0.882122722253417) (LR:  0 ) 
cost epoch50: -0.941513966858845 (cv cost: -0.883870995252808) (LR:  0 ) 
cost epoch60: -0.941531550554463 (cv cost: -0.884766616447599) (LR:  0 ) 
cost epoch70: -0.941535282199882 (cv cost: -0.884683810969873) (LR:  0 ) 
cost epoch80: -0.941536111546707 (cv cost: -0.884667962089245) (LR:  0 ) 
  [test trgainunder OK]
cost epoch90: -0.941536176001983 (cv cost: -0.884660973769204) (LR:  0 ) 
  [test trgainunder OK]
cost epoch100: -0.941536176001983 (cv cost: -0.884660973769204) (LR:  0 ) 
  [test trgainunder OK] [test cvgainunder OK]
[2 / 2 cond. OK to exit]
   dim X: [4,135]
   dim W1: [10,4] (min|max: -3.36208612220958, 2.81065207489658)
   dim bB1: [10,1] (min|max: -0.0102549001533971, 0.0551723214287623)
   dim W2: [1,10] (min|max: -0.0349268520031114, 0.0131917208210108)
   dim bB2: [1,1] (min|max: 0.0472983758883083, 0.0472983758883083)
   dim Y: [1,135]

It seems to me that in your vignette too little attention is paid to the objective function. After all, an objective function is what distinguishes your package from others, I personally chose your package for this very reason.

Thank you for your help.. Also, the question is off topic. will it CNN and RNN ?

[aboulaboul]

1) c(yhat) , c(y) -> Brillant ! 2) "..too little attention is paid to the objective function..." do you mean the custom cost function part is not enought explained ? 3) concerning CNN and RNN, sure for now that I won't do it alone ... But you're welcome to handle it ;-)

[mytarmail]

1) thanks :) 2) I think yes 3) Unfortunately, I am not a mathematician or a programmer, I have not written any machine learning algorithms, and in general I have little understanding of how they work. I'm just a user with below average skills at the moment.

Your package has an advantage over 99.5% of other machine learning packages 1) fitness function 2) multi-target and multi-output don't take it away

Thank you for your hard work and time