Details about training

[GinGinWang]

Hi, I am replicating your experiment's results. Could you give more the following details about the training on mnist and cifar10 which will replicate the results on your paper?

• lambda (trade-off between llk and reconstruction loss), is it fixed during training
• batch size
• optimizer, the learning rate
• when to stop training? If I use the validate set, which needs to be monitored? reconstruction loss, or negative llk, or novelty score to monitor

Thanks a lot! Best, Gin

[mathue]

Hi Gin,

from the supplementary material of the paper (chapter 7), one can infer lambda = 1 batch size = 256 learning rate = 10^-4 Optimizer = adam They stopped their training after a fixed number of 200 epochs (see closed issue) and monitored the LSA-loss (reconstruction loss plus autoregression loss (neg. llk)). However, if I train with these parameters, I only achieve an average AUROC-NS of 0.817 and 0.916 with a batch size of 64. Thus, they probably change some of these parameters for a better convergence. With batch size 256 it does not converge well but maybe I made a mistake. I attached my code for training (I changed the val() function in the mnist.py code to extract the first 90 % of the training data instead of the last 10 %). train.py.zip

[DavideA]

Hi and thank you for your interest.

@GinGinWang as correctly pointed out by @mathue, training details are provided in the supplementary material.

@mathue at a first glance, I can see a major issue in your training code:

• you are training on the validation split. The val function in the dataset implements the validation split, which is performed by shuffling indexes and taking the last 10%. You should implement a train function that puts the dataset into training mode (by taking the first 90% of training indexes).

Hope this helps, D

[mathue]

Hi Davide,

thanks alot for your reply. You are right, that's a shame, I made a mistake and trained on the 10 % validation set. When training on the 90 % training set for 200 epochs, I also achieved 0.972 AUROC-NS which is very close to what you reported in the paper.

Best regards Martin.

[GinGinWang]

Hi Davide, Sorry to bother you again. I replicate your experiments on cifar10 using all the parameters you recommended in your paper. After 200 epochs, I got AUROC 0.6. I also tried more epochs (1000 epochs), the AUROC did not increase. But I could have good results on the mnist dataset ... Is there any other training trick that is not mentioned in your paper? Thanks a lot for your help!