Interaction Blacklist for GBMs

[exalate-issue-sync[bot]]

We would like an “interaction blacklist” feature added to the GBM function. The goal is to prevent certain variables from interacting (i.e. if you blacklist variable X, allow X to enter only at the root and disallow all other variables to enter once X has entered).

More Details: Salford Systems has developed an Interaction Control Language (ICL) for TreeNet models. The language allows the modeler to specify precisely the types of interactions which will be permitted in the TreeNet for example you could write {code}ICL allow X1 X2 X3 X4 / 2{code} which specifies that only 2-way interactions are allowed among the collection of predictors listed (X1-X4) The ICL language is used for interaction detection work

You can also disallow interactions (which is what this feature request is asking for) {code}ICL ADDITIVE X1 X2 X3{code} the ADDITIVE keyword prevents any predictor from interacting with any other variable in the model practically +this means that should such a predictor be selected to split the root node of the tree than it can be the only predictor anywhere in that tree.+ NOTE: TreeNets restricted to ADDITIVE predictors can still contain trees with as many terminal nodes as the modeler prefers to work with, but each tree will be grown with a single predictor

[exalate-issue-sync[bot]]

Avkash Chauhan commented: #90308 (https://support.h2o.ai/helpdesk/tickets/90308) - H2O Feature Request - Interaction Blacklist for GBMs

[exalate-issue-sync[bot]]

Lauren DiPerna commented: this is a feature that exists for TreeNet: https://www.salford-systems.com/products/treenet

community pull requests appreciated

[exalate-issue-sync[bot]]

Michal Malohlava commented: Arno, what do you think about this one?

[exalate-issue-sync[bot]]

Arno Candel commented: So if you pick X to split the root, the entire rest of the tree must be split based on X? So a tree either has no splits on X or only splits on X?

[exalate-issue-sync[bot]]

Lauren DiPerna commented: [~accountid:557058:3bc534f4-c129-4d5f-b8c1-5a69d34942ee] yes that is correct.

[exalate-issue-sync[bot]]

Michal Malohlava commented: [~accountid:557058:eac185dd-5a5c-46e9-bb5a-13217ee9c218] pls attach more details about the required feature

[exalate-issue-sync[bot]]

Lauren DiPerna commented: what this feature request allows for: 1) How much lift do I get by having a feature interact with the other features compared to when I have that feature behave purely additively 2) how to limit interactions when doing residual modeling for a classification problem.

useful presentation from tree net: http://nymetro.chapter.informs.org/prac_cor_pubs/04-09%20Dan%20Steinberg%20Interaction%20Detection_3b.pdf

"To enforce literal non-interactivity we must rely on the ICL mechanism. If we specify that Xi is to enter the TreeNet as ADDITIVE then if we split the root using the MVI (missing value indicator) for Xi any subsequent split can use only Xi"

Dan Steinberg On Interaction Detection With TreeNet

Recent advances in machine learning technology make it possible to determine definitively whether or not interactions of any degree need to be included in a predictive model.

We can thus establish conclusively, for example, for a given set of predictors, that an additive model (one with no interactions) cannot be improved upon with interactions. Or alternatively, one might prove that a model with interactions will outperform a model without them.

Further, we can now identify precisely which interactions are supported by the data, and also the degree of interaction, even in very high dimensional data. The tools we use to achieve these results are extensions of Stanford University Professor Jerome Friedman's TreeNet, developed by the authors and embedded in the Salford Systems TreeNet 2.0 Pro Ex product. Steinberg illustrates the concepts in the context of a real world regression model where we are quickly able to identify all the important interactions with a modest number of boosted tree ensemble models.

http://library.fora.tv/2009/06/10/Dan_Steinberg_on_Interaction_Detection_with_TreeNet

[exalate-issue-sync[bot]]

Lauren DiPerna commented: if X1 is not allowed to interact with X2 that means that if the tree splits on X1 then it cannot split on X2 at any point from there on out. Think of it as, "if certain conditions are already meant in the tree what am I still allowed to do"

[exalate-issue-sync[bot]]

Lauren DiPerna commented: [~accountid:557058:389d9607-5bd8-4611-8c6a-755fe9295223] I've added the extra details for this ticket (both in description, and with resources in the comments). Should I close this ticket for myself, or do you want to reassign it now that this portion is taken care of? let me know if more detail is needed.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-3952 Assignee: Michal Malohlava Reporter: Avkash Chauhan State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A