igrigorik / decisiontree

ID3-based implementation of the ML Decision Tree algorithm
1.44k stars 130 forks source link

Why are gains normalized at each node? #18

Closed nicomahler closed 10 years ago

nicomahler commented 10 years ago

Thank you for your great gem, Ilya!

We are currently working on a project, where we need to evaluate the importance of each attribute for a given decision tree. This is done by summing over the tree the gain for each attribute.

We observe that the results we get are quite different than results that are produced using the rpart package in R and the scikit-learn package in Python. This is explained by the fact that the gain is normalized at each node in the decisiontree gem (here and here).

Hence our question: is there any reason for normalizing the gain at each node in the decisiontree gem?

If not, we would like to make a pull request for computing a non-normalized gain - and, maybe later, for adding a function, which would compute the importance of each attribute.

igrigorik commented 10 years ago

image

Courtesy of http://books.google.com/books?id=GlKIIR78OxkC&pg=PA56&lpg=PA56&dq=decision+tree+and+normalizing+gain&source=bl&ots=0_5IfR-eTT&sig=WotuR6rtcA-Dq51VpF9dts7dWXM&hl=en&sa=X&ei=lv9HU42hK-OU8QHTl4HwDg&ved=0CE8Q6AEwAw#v=onepage&q&f=false