TabTransformer implementation in study 🤖

[KarelZe]

Dear @kathrinse,

thanks for your excellent study on deep neural networks for tabular data! I came across your repository and paper during the research for my on-going master's thesis.

I noticed two aspects that I wanted to share with you:

• You base your implementation of the TabTransformer on this repository. The implementation, however deviates from the paper (http://arxiv.org/abs/2012.06678) concerning the column embedding, which is one of the author's contributions. The paper proposes a column embedding that consists of feature-value-specific embedding and a shared embedding. Both get concatenated or added element-wisely. The shared embedding is important in the author's work (see p. 10-11). The used implementation doesn't implement any shared embedding. Also, the implementation introduces a scaling factor for the hidden dimensions in input_size // 8 (see https://github.com/kathrinse/TabSurvey/blob/main/models/tabtransformer.py) of the MLP, that I could not find in the paper (see p. 3). Thus, the net might have a much smaller capacity.
• Regarding the calculation of attention maps e. g., for TabTransformer you follow (http://arxiv.org/abs/2106.01342) and sum over the attention heads from the first transformer block (see e. g., https://github.com/kathrinse/TabSurvey/blob/main/models/tabtransformer.py). Several papers (see e. g., https://arxiv.org/abs/1905.09418 or https://arxiv.org/abs/1905.10650) from admittedly from the non-tabular domain suggest that attention heads have different importance, and so do other Transformer blocks. Thus, I wonder if the current approach is optimal. I adopted an alternative method proposed in (http://arxiv.org/abs/2103.15679), which could also be interesting for you. Thereby, one can weigh all heads and incorporate all Transformer blocks.

Sorry if I have missed something.

Keep up the excellent work :100:

Best, Markus