boennecd
commented
7 years ago I have made a few additions. Sorry for doing this so shortly after I opened the request. The changes are:
gpe_tress(sorry for the typo but I have kept this to still make the previous examples run-able) now uses gradient boosting for binary outcomes whenpartykit::ctreeis usedgpe_earthto uses gradient boosting for binary outcomes- Added digits option to
print.pre - Changed the way the arguments to
cv.glmnetis passed. The new setup allows the user to use other functions thencv.glmnet
Here is an example of the changes. The Sonar data set is binary. Notice both the computation time and mean cv error. The computation using gradient boosting may provide the answer to the comments I pose in fdc23a2b38afb50f1e87d6a3b07ea1122856a023
#####
# Install version
devtools::install_github(
"boennecd/pre@34845cb30fb3f20847f0c8eca076d878eebd6ad9")
# Load and prepare data
data(Sonar, package = "mlbench")
dat <- Sonar
#####
# Fit w/ model as in Friedman & Popescu (2008) using partykit::glmtree
set.seed(seed <- 4455668)
system.time(fit <- gpe(
Class ~ ., data = dat,
base_learners = list(gpe_tress(ntrees = 100), gpe_linear())))
#> user system elapsed
#> 12 0 12
options(digits = 4) # used to alter print
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.02065
#> number of terms = 20
#> mean cv error (se) = 0.6228 (0.06798)
#>
#> [output abbreviated]
options(digits = 2) # use fewer digits
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.021
#> number of terms = 20
#> mean cv error (se) = 0.62 (0.068)
#>
#> [output abbreviated]
#####
# Fit model w/ MARS terms w/ trees using partykit::glmtree
set.seed(seed)
system.time(fit <- gpe(
Class ~ ., data = dat,
base_learners = list(gpe_tress(ntrees = 100), gpe_linear(), gpe_earth())))
#> Beware that gpe_earth will use gradiant boosting
#> user system elapsed
#> 13.98 0.01 14.00
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.017
#> number of terms = 27
#> mean cv error (se) = 0.56 (0.043)
#>
#> [output abbreviated]
#####
# Fit model w/ MARS terms and trees using partykit::ctree w/ gradient boosting
set.seed(seed)
system.time(fit <- gpe(
Class ~ ., data = dat,
base_learners = list(gpe_tress(ntrees = 100, use_grad = TRUE),
gpe_linear(), gpe_earth())))
#> Beware that gpe_earth will use gradiant boosting
#> user system elapsed
#> 5.00 0.02 5.02
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.011
#> number of terms = 43
#> mean cv error (se) = 0.59 (0.042)
#>
#> [output abbreviated]
#####
# Fit model w/ MARS terms and trees using partykit::ctree w/ gradient boosting
# w/ higher learning rate
set.seed(seed)
system.time(fit <- gpe(
Class ~ ., data = dat,
base_learners = list(gpe_tress(ntrees = 100, use_grad = TRUE, learnrate = .5),
gpe_linear(),
gpe_earth(learnrate = .5))))
#> Beware that gpe_earth will use gradiant boosting
#> user system elapsed
#> 4.8 0.0 4.8
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.01
#> number of terms = 45
#> mean cv error (se) = 0.52 (0.06)
#>
#> [output abbreviated]
#####
# Fit model w/ MARS terms and trees using partykit::ctree w/ gradient boosting
# w/ higher learning rate and elastic net instead L1 yielding a less sparse
# solution
set.seed(seed)
system.time(fit <- gpe(
Class ~ ., data = dat,
base_learners = list(gpe_tress(ntrees = 100, use_grad = TRUE, learnrate = .5),
gpe_linear(),
gpe_earth(learnrate = .5)),
penalized_trainer = gpe_cv.glmnet(alpha = 0.5)))
#> Beware that gpe_earth will use gradiant boosting
#> user system elapsed
#> 5.02 0.03 5.40
fit
#>
#> Final ensemble with cv error within 1se of minimum:
#> lambda = 0.0095
#> number of terms = 164
#> mean cv error (se) = 0.42 (0.055)
#>
#> [output abbreviated]
Main addition
Added the
gpefunction to make more general sparse prediction ensemble. The advantage is that:1) It is easy to generalize to other methods of getting basis function 2) The implementation is modular so it is easy to test each part 3) An option to uses basis function from products of hinge function through the MARS method is added. It uses the
earthpackageHere is an example of the new function is given below. We use the Boston housing example also used in Friedman & Popescu (2008):
The new function is documented so further information about the setup is in
?gpeafter installation e.g. by calling:The se on the error seems to be smaller in the (few) data sets have looked when the MARS terms are used. See commit fe984578a2195804a8bd7c920c0e17ce52267b0a for another example
I can add an example with
gpeto the readme if you wantOther major changes
CRAN and tests
I fixed the issues with tests that did not pass with
devtools::check(document = FALSE, cleanup = FALSE). The cause and solution is described in commit 4f4bc47d86a9da7fa3694b0fff4056b5273e7601I strongly suggest to keep the tests with the builds. This will cause them also to run on CRAN which can discover bugs on different platforms. The new version does include the tests in the build
README
I have changed the formatting of the readme using
knitr. You can see the new readme with plots on https://github.com/boennecd/pre/tree/b5d54f64a0243083b07e5b458b54f72f541df881Subsampling
Notice that
gpe_sampledoes make replacement whensample_frac < 1and the weights are unequal. The motivation is that the weights will not be representative otherwise. I have not changed the setup inprebut suggest to do similarly as ingpe_sampleFactors
Fixed bugs with factor levels when the learning rate is postive in
pre. See commit aebf7d6af1f7582d743408f771f84d9b05f9d262Removing complements
Changed the method to remove complement rules to a considerably faster methods that provides the same result. See commit 23eb1cd7829d3e76af86e20ead83c57e005b46ee
Other remarks
The
partykit::glmtreeis terribly slow for larger data sets. I have not found a solution yet but have posted some options in commit fdc23a2b38afb50f1e87d6a3b07ea1122856a023Minor changes
bsnullmodcallwill change theverboseargument of$callto the argument passed tobsnullmodcallFixed typos usingdevtools::spell_checkUpdated test that did not pass when I downloaded the new version of the package. See commit fdba929c08b56dbb38015b10cb98e7c6fa384022