Trying to reproduce some result from the paper

yandex-research / rtdl-num-embeddings

(NeurIPS 2022) On Embeddings for Numerical Features in Tabular Deep Learning

https://arxiv.org/abs/2203.05556

MIT License

312 stars 34 forks source link

Trying to reproduce some result from the paper #21

Open IlyaTrofimov opened 2 weeks ago

IlyaTrofimov commented 2 weeks ago

Hi!

I'm trying to reproduce numbers from Table 4 in https://arxiv.org/pdf/2203.05556. For California Housing there should be 0,439 for MLP-T-LR. When I use hyperparameter from https://github.com/yandex-research/rtdl-num-embeddings/blob/abf8a8b35854e4b06476bb48902096b0b58ffce2/exp/mlp-t-lr/california/0_tuning/report.json

I get around ~0.45 at test dataset for various seeds. Can you please spot the problem?

Yura52 commented 6 days ago

Hi! Unfortunately, I cannot replicate the environment on my current machine to confirm the issue. Have you tried retuning the hyperparameters from scratch, and evaluating the newly tuned hyperparameters? Sometimes, tuning hyperparameters in one environment and evaluating them in a different environment can result in metric fluctuations.