Failed to reproduce the results presented in the paper

[zzzhezeng]

Hello,

The approach and results presented in the paper "Subspace Inference for Bayesian Deep Learning" impress me a lot and I would like to reproduce the experiments, specifically the UCI experiments, using this repository.

I followed the instructions and process described in the README and managed to get things run. However, I failed to obtain similar numbers for certain datasets.

To be more precise:

• versions of libraries match.

• the datasets are downloaded from Google Drive as suggested in the README under drbayes/experiments/uci_exps.

• all experiments on small UCI datasets are launched by the script run_ucismall.sh under drbayes/experiments/uci_exps/bayesian_benchmarks/tasks.

• the reproduced unnormalized test likelihoods for the yacht dataset are significantly different from those reported in the paper as shown below.

	paper result	reproduce result
PCA+ESS (SI)	-0.225 ± 0.400	-2.493 ± 0.067
SWAG	-0.404 ± 0.418	-2.545 ± 0.053

I would be extremely grateful if anyone could suggest what I could've done wrong in my reproduction experiments.

Thank you in advance!

[wjmaddox]

Ugh, yeah it looks like this was on our end (see also the longer email) and also luckily only yacht rather than any other datasets. I spent a while re-tuning yacht and it looks like it was only the weight decay that was off. Switching to a lower weight decay of 1e-3 provides a more similar LL and RMSE result:

edit: i'll fix the script in the code tomorrow.