search

dreamquark-ai / tabnet

PyTorch implementation of TabNet paper : https://arxiv.org/pdf/1908.07442.pdf
https://dreamquark-ai.github.io/tabnet/
MIT License
2.61k stars 485 forks source link

feat: enable feature grouping for attention mechanism #443

Closed Optimox closed 1 year ago

Optimox commented 1 year ago

What kind of change does this PR introduce? This PR solves #122 in a new way.

Embeddings from a same column are automatically grouped together. Now attention works at the group level and not feature level. So without specifying anything groups for different categorical features are created.

This is a new as it allows users to specify features they would like to be grouped together by the attention mechanism. This can be very useful when sparse features are created (for example after a TD-IDF), attention has a hard time using this kind of features because of sparsity coming from both the data and the attention. Now you can group all those features together and have a single attention.

Does this PR introduce a breaking change? I'm not sure if this should be considered a breaking change or not. I think so as old trained models used with this new code will have a different behaviour.

What needs to be documented once your changes are merged?

I think all changes are already documented in this PR.

Closing issues closes #122

gauravbrills commented 11 months ago

can we add an example of how to use this , bit confused about this @Optimox

Optimox commented 11 months ago

@gauravbrills There is an example on this notebook : https://github.com/dreamquark-ai/tabnet/blob/develop/census_example.ipynb

You simply need to give a list of groups, each group is a list with the index of the features forming the group. Attention mechanism will consider each group as one single feature, so all features of the group will get the same attention (and importance). Note that all embedding dimensions generated by a categorical feature will be grouped together.