For the next release we should spell out more of computational details behind exact search. We should communicate the following better:
a) exact search is only feasible (and intended) for "moderately" (to be defined) sized data
b) exact search is intended for shallow (i.e. interpretable) trees (depth = 2 or 3 max)
c) spell out more what the amortized run time means in practice. We have a table in https://github.com/grf-labs/policytree/issues/46#issuecomment-686658293, this could be part of the docstring as a point of reference
d) + spell out what the cardinality of the Xs means for the runtime: i.e. benefits split.step + the increased benefit of rounding covariates (which involves less point insertions in the algo)
We should also add some suggested heuristics:
e) to decrease p you can include only variable's who variable_importance passes some threshold
f) to asses the stability of a policy / avoid fitting one on massive data, you can fit several policies on smaller manageable subsets and compare the reward.
Another peripheral docstring improvement:
g) tree search is unconstrained, but you can bake in a constant cost offset by subtracting it from the objective. This should be an example in the docstring.
Let us make sure @susanathey and @swager agrees that we are in fact making things clearer.
For the next release we should spell out more of computational details behind exact search. We should communicate the following better:
a) exact search is only feasible (and intended) for "moderately" (to be defined) sized data b) exact search is intended for shallow (i.e. interpretable) trees (depth = 2 or 3 max) c) spell out more what the amortized run time means in practice. We have a table in https://github.com/grf-labs/policytree/issues/46#issuecomment-686658293, this could be part of the docstring as a point of reference d) + spell out what the cardinality of the Xs means for the runtime: i.e. benefits split.step + the increased benefit of rounding covariates (which involves less point insertions in the algo)
We should also add some suggested heuristics:
e) to decrease p you can include only variable's who variable_importance passes some threshold f) to asses the stability of a policy / avoid fitting one on massive data, you can fit several policies on smaller manageable subsets and compare the reward.
Another peripheral docstring improvement: g) tree search is unconstrained, but you can bake in a constant cost offset by subtracting it from the objective. This should be an example in the docstring.
Let us make sure @susanathey and @swager agrees that we are in fact making things clearer.