About the feature shapes of embeddings in MLP vs FT_Transformer

[yiqings]

Hi, thanks for the codes and insightful paper "On Embeddings for Numerical Features in Tabular Deep Learning".

We few a bit confused about the feature shapes of the embeddings in MLP:

• In FT_Transformer, suppose the input shape is [batch_size,number_of_numerical_features] (WLOG omit considering categorical features). Then, the embeddings turn to be [batch_size,number_of_numerical_features,dimension_of_embeddings] (i.e., ndim=3) with NumEmbeddings.
• However, if it is directly generalized to MLP with ndim=3 for embeddings. Then the output shape after multiple MLP blocks turn to be [batch_size, number_of_numerical_features, dimension_of_mlp_output] (also ndim=3). Is there any pooling strategies applied to derive the logits, or if the embeddings are tailored to ndim=2 specifically for MLPs?

CC: @sxjscience

Thanks very much.

[Yura52]

Hi! Thanks for your interest in the project!

For all backbones (MLP, ResNet, Transformer), the shape of the tensor after the embedding operation is [batch_size,number_of_numerical_features,dimension_of_embeddings], i.e. ndim=3.

However, for MLP and ResNet, the class FlatModel is selected here, so the embeddings are reshaped to ndim=2 before passing to the model, as can be seen here.

[yiqings]

Thank you @Yura52 so much for the prompt rely, and it helps a lot!