PyOD implementations show lower results compared to the paper, and to the corresponding sklearn scores

[kordc]

I was trying to explore the PyOD functionality, and I can see, that the results by default are very low compared to the ADBench paper. Moreover, when I compare the Isolation Forest, the sklearn implementation just performs better. I used fixed random_state so you can reproduce it.

Isolation Forest

from pyod.models.iforest import IForest
data = np.load('../data/numerical/01_breastw.npz',
                   allow_pickle=True)  # very simple dataset
X, y = data['X'], data['y']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, shuffle=True, random_state=23)

isf = IForest(random_state=23).fit(X_train)
isf_pred = isf.predict(X_train)
roc_auc_score(y_train, isf_pred) # => 0.6443854458530086

The result is 0.6407. Complared to the sklearn:

from sklearn.ensemble import IsolationForest
data = np.load('../data/numerical/01_breastw.npz',
                   allow_pickle=True)  # very simple dataset
X, y = data['X'], data['y']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, shuffle=True, random_state=23)

isf = IsolationForest(random_state=23).fit(X_train)
isf_pred = isf.predict(X_train)
isf_pred_shifted = isf_pred.copy()
isf_pred_shifted[isf_pred_shifted == 1] = 0
isf_pred_shifted[isf_pred_shifted == -1] = 1
roc_auc_score(y_train, isf_pred_shifted) # => 0.956080882497012

The above result of 0.956 corresponds more to the paper, where score is equal to 0.9832

ECOD

from pyod.models.ecod import ECOD
data = np.load('../data/numerical/01_breastw.npz',
                   allow_pickle=True)  # very simple dataset
X, y = data['X'], data['y']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, shuffle=True, random_state=23)

isf = ECOD().fit(X_train)
isf_pred = isf.predict(X_train)
roc_auc_score(y_train, isf_pred) # => 0.6490683229813665

The above 0.649 doesn't correspond to the paper's 99.17.

HBOS

from pyod.models.hbos import HBOS
data = np.load('../data/numerical/01_breastw.npz',
                   allow_pickle=True)  # very simple dataset
X, y = data['X'], data['y']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, shuffle=True, random_state=23)

isf = HBOS().fit(X_train)
isf_pred = isf.predict(X_train)
roc_auc_score(y_train, isf_pred) # => 0.6443854458530086

The above 0.644 doesn't correspond to the paper's 98.94.

I use the PyOD in the simplest possible manner, and I checked the dataset with the isolation forest. That's why I think something is wrong.

The dataset is available in the ADBench repository - breastw

[kordc]

@yzhao062 could you say which version of PyOD you used in the ADBench? I tried release 1.0.1 and it also has those results visible above.

[yzhao062]

isolation forest is a bit weird since I did not implement it but imported it from scikit-learn

see here: https://github.com/yzhao062/pyod/blob/master/pyod/models/iforest.py

from sklearn.ensemble import IsolationForest

One reason is that the hyperparameter setting and some built-in randomness of iforest.

[yzhao062]

according to adbench (https://github.com/Minqi824/ADBench), we use pyod 1.0.0 for this purpose. One thing to note is, we do not predict labels but predicts raw outlier scores for ROC...

y_pred should be continuous scores, and y should be binary labels for ROC.

[kordc]

That's an important note, I'll rerun my experiments in this manner, thanks!

[kordc]

This explains everything, results are now reproducible. Thank you for the help!