Need help on reproducing KDD result

[jcbnose]

Trying to replicate the KDD result on paper - (i.e. cross-e: P/R/F1 = 0.9200/0.9582/0.9372) Ran model as-is using: [bigan, kdd, w=0.1, m='cross-e', d=1, rd=42] First 100 epoch got great result: Epoch 97 | time = 15s | loss gen = 36.8243 | loss enc = 391.7180 | loss dis = 0.2705 Epoch 98 | time = 17s | loss gen = 39.3836 | loss enc = 418.4268 | loss dis = 0.2501 Epoch 99 | time = 15s | loss gen = 34.9579 | loss enc = 437.5724 | loss dis = 0.1583 Testing evaluation... Testing : Prec = 0.9434 | Rec = 0.9584 | F1 = 0.9509

And ran another 100 epoch got worse, losses didn't seem to converge (assuming the code picked up the model where it left off) Epoch 97 | time = 15s | loss gen = 1.0412 | loss enc = 91.7826 | loss dis = 1.0371 Epoch 98 | time = 15s | loss gen = 1.0412 | loss enc = 94.2479 | loss dis = 1.0378 Epoch 99 | time = 17s | loss gen = 1.0432 | loss enc = 97.2879 | loss dis = 1.0332 Testing evaluation... Testing : Prec = 0.3215 | Rec = 0.3266 | F1 = 0.3240

Questions: 1) May I know what may have gone wrong? I downloaded the code and ran it as is. I also ran for 1000 Epoch but result was not good at all. I once even ran it for 5000 epoch and got this result: Epoch 4999 | time = 15s | loss gen = 3.2600 | loss enc = 329.4751 | loss dis = 0.3632 Testing : Prec = 0.1924 | Rec = 0.5634 | F1 = 0.2869 2) How come the 2nd run, the loss gen, loss enc were much smaller but F1 got worse? Shouldn't be the lower the losses, the better F1? (given than: list_scores = (1-weight) gen_score + weight dis_score) 3) Why the loss enc always seem so big? Anyway to improve it? Note that I tried to add a layer_3 in encoder and/or to add dropout in encoder and generator but didn't help.

Thanks

[houssamzenati]

Thanks for your interest in our research,

1) It's good you managed to reproduce the results. Do try to replicate them with the same number of random seeds as we did (ref. our prepublished paper for the details) and with the variance as well. 2) We followed the same methodology as the original DAGMM paper (Zong et al, 2018) and DSEBM paper (Zhai et al, 2016). As their code is not open source, we contacted them multiple times via emails to insure our setup was exactly the same. This also includes the model finetuning and hyperparameters search methodology. You may also email them and have their advise. 3) There is no such straightforward link between the training losses and the performances of the model. As a matter of fact, the training losses are adversarial (ref original BiGAN paper) whereas our scoring loss is derived from the AnoGAN paper (which is our main baseline) which are reconstruction losses. 4) For the architecture search, an empirical observation (especially because there are not so many papers on GANs with non image data, and so architectures are scarce) we made was that it was always better to find a GAN architecture, finetune it, and then build on it to find a BiGAN architecture. You may have a look at the architecture details in the appendix of our prepublished paper.

Thanks

[jcbnose]

Thanks for your reply. Really appreciate it. 1a. I put the arguments (including random seeds) on the second line of my original post. I believe I used the same parameters. What I don't understand is why after first 100 epochs the results was good but when it picked up and trained for more epochs it became much worse. 1b. when you said prepublished paper, did you mean this [ https://arxiv.org/abs/1802.06222 ]? if not, how can I access it?

2. will do
3. I am interested to know if you can recall your observations during training, how big the losses were and whether they converged, with how many epochs?
4. thanks for your suggestions.

[houssamzenati]

My pleasure,

1a. To understand this, there are so many possible issues. I don't remember the properties of the dataset. But some categories of "attacks" (which are treated as "normal" in this task -> please see the original papers and ours for the explanation of this choice...) may be more represented in the data than other "attacks". Therefore the model may overfit one or another category when you train it for too long... In general I would really recommend to look at this very dataset properties and the BiGAN/AliGANs training trends in other datasets (e.g image datas) to find possible explanations 1b. Yes

2. Cool
3. I don't really remember, but what were your observations on MNIST?
4. Cool