Unable to reproduce results for DAGMM

[jonomon]

Hello,

Thank you for the nice library!

I was just wondering if you managed to reproduce the results in Zong, Bo, et al. "Deep autoencoding gaussian mixture model for unsupervised anomaly detection." International conference on learning representations. 2018.

I used the following configuration:

DAGMMConfig(gmm_k=2, hidden_size=4, num_epochs=20000, lr=0.0001, batch_size=1024)

and only managed to get the following results on the Thyroid dataset (.mat obtained from http://odds.cs.stonybrook.edu):

Precision: 0.0238
Recall: 0.3571
F1: 0.0447

[aadyotb]

Hi @jonomon, there can be some subtle differences in the way precision/recall are computed, the way the detection threshold is chosen, and how the model handles point data (which the Thyroid dataset is) vs time series data. Before anything else, you should try to use PointwisePrecision, PointwiseRecall, and PointwiseF1 as your evaluation metrics, as the default ones are specialized for time series data. If this doesn't resolve the issue, @yangwenzhuo08 can you answer any further questions?

[jonomon]

Hi @aadyotb

Thank you for the reply.

Using PointwisePrecision, PointwiseRecall, and PointwiseF1 had the following results:

Precision: 0.0238
Recall: 0.3409
F1: 0.0446

It seems like it help too much.

As a side note, both the Autoencoder and VAE achieved comparable results on the Thyroid dataset out of the box.

Model name: Autoencoder
Precision: 0.4444
Recall: 0.3636
F1: 0.4000

Model name: VAE
Precision: 0.4242
Recall: 0.3182
F1: 0.3636