Regarding Position Encoding

[isCopyman]

The paper did not mention a specific location encoding method. I took a brief look at the code. May I ask if a learnable encoding method is being used.

[Raphaaal]

Hi,

Thank you for your interest in this work.

Yes, we used two standard learnable positional embeddings (BERT-like), one for the row position and one for the column index.

That is to say, a vector is learned for each position throughout training and added element-wise to the tokens embeddings.

These positional embeddings are initialized at the following locations:

• FT-Transformer: https://github.com/Raphaaal/fieldy/blob/8d5e39f9e5d8386c3065c5bc0ffb589d134b99b7/models/models.py#L191

• Tabbie: https://github.com/Raphaaal/fieldy/blob/8d5e39f9e5d8386c3065c5bc0ffb589d134b99b7/models/models.py#L360

• TabBERT (column-based): https://github.com/Raphaaal/fieldy/blob/8d5e39f9e5d8386c3065c5bc0ffb589d134b99b7/models/models.py#L535 handled in the ColumnTabBERTEmbeddings and in the CustomBertForMaskedLM modules.

• TabBERT (row-based): https://github.com/Raphaaal/fieldy/blob/8d5e39f9e5d8386c3065c5bc0ffb589d134b99b7/models/models.py#L629 handled in the RowTabBERTEmbeddings and in the CustomBertForMaskedLM modules.

• Fieldy: https://github.com/Raphaaal/fieldy/blob/8d5e39f9e5d8386c3065c5bc0ffb589d134b99b7/models/models.py#L728 handled in the CustomBertForMaskedLM module.