what are the inputs to the encoder and decoder of the Transformer model during pre-training

[Sakura5540]

Hello team

I'm sorry I wasn't clear in my last question.My question is, what are the inputs to the encoder and decoder of the Transformer model during pre-training, respectively? I guess, the encoder's input is the features of the dataset, but I don't know what the decoder's input will be?

Thank you~

[noahho]

The encoders inputs are the synthetic datasets (see the synthetic datasets here: https://github.com/automl/TabPFN/blob/main/tabpfn/PriorFittingCustomPrior.ipynb), the decoder inputs are the encoded datasets after they were encoded by the transformer. The loss is applied to the target values of the synthetic datasets (see same notebook).

All the best