Anomaly GAN Implementation Seems to be Flawed

[John-Almardeny]

The AnoGan model is not tested in PyOD, nor is an example provided, and when trying it locally it throws a cascade of "internal" errors. The test module is commented out. Apparently, there is a problem with this model yet it was released in the new version!

An example out of many:

model = AnoGAN(epochs=10, contamination=contamination)
model.fit(x_train)

Results in: KeyError: 'The optimizer cannot recognize variable dense_26/kernel:0. This usually means you are trying to call the optimizer to update different parts of the model separately. Please call optimizer.build(variables) with the full list of trainable variables before the training loop or use legacy optimizer `tf.keras.optimizers.legacy.{self.class.name}.'

[mbongaerts]

Dear @John-Almardeny,

The AnoGAN implementation is automatically tested on multiple platforms when doing a pull request. You can see the results here: appveyor

So, I guess there is something wrong with your TensorFlow installation?

I just installed the newest version of PyOD and ran the following code:

contamination = 0.1  # percentage of outliers
n_train = 200  # number of training points
n_test = 100  # number of testing points

# Generate sample data
X_train, X_test, y_train, y_test = generate_data(
    n_train=n_train,
    n_test=n_test,
    n_features=10,
    contamination=contamination,
    behaviour="new",
    random_state=42,
)

# train AnoGAN detector
clf_name = 'AnoGAN'
clf = AnoGAN(activation_hidden='tanh',
             dropout_rate=0.2,
             latent_dim_G=2,
             G_layers=[10, 50],
             verbose=1,
             D_layers=[50, 10],
             index_D_layer_for_recon_error=1,
             epochs=1000,
             preprocessing=False,
             learning_rate=0.001,
             learning_rate_query=0.01,
             epochs_query=20,
             batch_size=32,
             output_activation=None,
             contamination=0.1)

clf.fit(X_train)

clf.plot_learning_curves( start_ind=0,  window_smoothening=10)

I see the labels in this plot are still not correct

# get the prediction labels and outlier scores of the training data
y_train_pred = clf.labels_  # binary labels (0: inliers, 1: outliers)
y_train_scores = clf.decision_scores_  # raw outlier scores

# get the prediction on the test data
y_test_pred = clf.predict(X_test)  # outlier labels (0 or 1)
y_test_scores = clf.decision_function(X_test)  # outlier scores

# evaluate and print the results
print("\nOn Training Data:")
evaluate_print(clf_name, y_train, y_train_scores)
print("\nOn Test Data:")
evaluate_print(clf_name, y_test, y_test_scores)

# # visualize the results
visualize(clf_name, X_train[:, 0:2], y_train, X_test[:, 0:2], y_test, y_train_pred,
          y_test_pred, show_figure=True, save_figure=True)

So, for me everything seems to work just fine.

[John-Almardeny]

Dear @mbongaerts , Thanks for your feedback.

1. I don't think there is anything wrong with my Tensoflow installation because I am using it with many different deep learning models I created (e.g. AutoEncoder, LSTM, CNN) and with PyOD deep learning models and I can confirm that I just tested them all and they all worked fine (except for this one of course).
2. Based on the PyOD protocol and requirements, all models must be accompanied by an example in the example directory and a set of unit tests in the test directory. AnaGAN, however, don't have those, and unit tests are commented out, why? The PyOD owner does not accept any model without these, and this is the first time I see such a huge exception (or violation of the rules) in PyOD.

Points number 1 and 2 naturally led me to think that this model is flawed, otherwise, there is a big valid why or what's going on.

[mbongaerts]

Dear @John-Almardeny,

Did you try to run my example above?

Your point 2 is weird indeed. My initial Pull request did contain all of this actually: https://github.com/yzhao062/pyod/pull/412/files#diff-2b5c140da123fa09f91ac1d6cf657fc6da9fabfb5f85aee6b72ce8f9fac3623a

Maybe @yzhao062 knows more?

[blackDZS]

@mbongaerts @John-Almardeny I meet this problem too when I run test_alad.py, and it reports the error

 KeyError: 'The optimizer cannot recognize variable dense_3/kernel:0. This usually means you are trying to call the optimizer to update different parts of the model separately. Please call `optimizer.build(variables)` with the full list of trainable variables before the training loop or use legacy optimizer `tf.keras.optimizers.legacy.{self.__class__.__name__}.'

and I go to keras's code find that the _var_key(varaible) and _index_dict is different

print(self.enc.optimizer._index_dict)

{'dense/kernel_7': 0, 'dense/bias_20': 1, 'dense_1/kernel_37': 2, 'dense_1/bias_50': 3, 'dense_2/kernel_67': 4, 'dense_2/bias_80': 5}

[self.enc.optimizer._var_key(_) for _ in self.enc.trainable_variables]

['dense_3/kernel_105', 'dense_3/bias_118', 'dense_4/kernel_135', 'dense_4/bias_148', 'dense_5/kernel_165', 'dense_5/bias_178']

obviously, the name in trainable_variables and optimizer is different.

I find that the code in alad.py, there are two optimizer and five model, the optimizer will be rebuild when compile to another model.

# Set optimizer
opt_gen = Adam(learning_rate=self.learning_rate_gen)
opt_disc = Adam(learning_rate=self.learning_rate_disc)

self.dec.compile(optimizer=opt_gen)
self.enc.compile(optimizer=opt_gen)
self.disc_xz.compile(optimizer=opt_disc)
self.disc_xx.compile(optimizer=opt_disc)
self.disc_zz.compile(optimizer=opt_disc)

I test the code below, and it can work normaly !

# Set optimizer
opt_gen = Adam(learning_rate=self.learning_rate_gen)
opt_enc = Adam(learning_rate=self.learning_rate_gen)
opt_disc_xz = Adam(learning_rate=self.learning_rate_disc)
opt_disc_xx = Adam(learning_rate=self.learning_rate_disc)
opt_disc_zz = Adam(learning_rate=self.learning_rate_disc)

self.dec.compile(optimizer=opt_gen)
self.enc.compile(optimizer=opt_enc)
self.disc_xz.compile(optimizer=opt_disc_xz)
self.disc_xx.compile(optimizer=opt_disc_xx)
self.disc_zz.compile(optimizer=opt_disc_zz)

and the same problem of anogan.py in these code