Open jcarlis3 opened 3 years ago
Jason, I am out until July 19th. I will take a look then.
Best, Jeff
Get Outlook for Androidhttps://aka.ms/ghei36
From: Jason Carlisle @.>
Sent: Thursday, July 1, 2021 10:17:30 AM
To: jeffreyevans/rfUtilities @.>
Cc: Jeffrey Evans jeffrey_evans@TNC.ORG; Mention @.***>
Subject: [jeffreyevans/rfUtilities] Error in rf.modelSel
(#7)
Hi @jeffreyevanshttps://github.com/jeffreyevans ,
I ran rf.modelSel back in 2017 in my sage-grouse research with Melanie et al. I'm dusting that analysis off, and Melanie pointed out that my MIR values should be 0-1, but several were negative. I see v2.2-0 has incorporated some bug fixes in that function, so I installed and attempted to re-run, but now get an error when running the updated rf.modelSel. I can reproduce the same error using the example in the help doc for rf.modelSel:
require(randomForest) require(rfUtilities)
sessionInfo()
#
#
#
#
#
data(airquality) airquality <- na.omit(airquality)
xdata = airquality[,2:6] ydata = airquality[,1]
( rf.regress <- rf.modelSel(airquality[,2:6], airquality[,1], imp.scale="se") )
( rf.regress <- rf.modelSel(airquality[,2:6], airquality[,1], method="Wright") )
ydata = as.factor(ifelse(ydata < 40, 0, 1))
( rf.class <- rf.modelSel(xdata, ydata, ntree=1000) )
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jeffreyevans/rfUtilities/issues/7, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACLKH75SC4JZVNBXVQ6IMATTVSIJVANCNFSM47VBNGPQ.
Brief recap of an off-thread discussion (in case others are having this same issue):
Negative MIR values are caused by the original importance values being negative. Suggestion is to use the CRAN version of the package (currently 2.1-5) and manually remove variables with negative MIR. Future developments may build in this sort of screening, but no fixes are imminent.
I am wondering what you would consider a reasonable behavior for negative importance values? Negative importance values are a valid possible outcome simply denoting that the random permutation worked better and resulted in no feature contribution to the model. The magnitude of the negative value is irrelevant, representing random variation and, mostly controlled by the stochastic element in the Bootstrap. As such, should negative importance simply be truncated at zero? I can add an argument that controls the behavior of negative values.
One huge advantage of the new implementation of the importance function is that it now allows for a significance test that allows one to evaluate the p-value of the feature contribution and screen-out insignificant parameters. It also supports a much faster and flexible implementation of random forests (through the ranger package).
If you are wanting to replicate a previous analysis and produce results consistent with your original model, please use the CRAN version and specify a consistent random seed to control stochasticity. I am still planning on getting back to the development version and submitting to CRAN but, life got in the way so this will not happen until the beginning of the year. If there is a specific function that you would like me to address I can find bandwidth to accommodate you, just let me know.
Best, Jeff
From: Jason Carlisle @.>
Sent: Monday, October 18, 2021 11:52 AM
To: jeffreyevans/rfUtilities @.>
Cc: Jeffrey Evans @.>; Mention @.>
Subject: Re: [jeffreyevans/rfUtilities] Error in rf.modelSel
(#7)
Brief recap of an off-thread discussion (in case others are having this same issue):
Negative MIR values are caused by the original importance values being negative. Suggestion is to use the CRAN version of the package (currently 2.1-5) and manually remove variables with negative MIR. Future developments may build in this sort of screening, but no fixes are imminent.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jeffreyevans/rfUtilities/issues/7#issuecomment-946017169, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACLKH7ZLB55VQDPT5BENS7TUHRNC5ANCNFSM47VBNGPQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Hi @jeffreyevans ,
I ran
rf.modelSel
back in 2017 in my sage-grouse research with Melanie et al. I'm dusting that analysis off, and Melanie pointed out that my MIR values should be 0-1, but several were negative. I see v2.2-0 has incorporated some bug fixes in that function, so I installed and attempted to re-run, but now get an error when running the updatedrf.modelSel
. I can reproduce the same error using the example in the help doc forrf.modelSel
: