Add support for scikit-learn models

[ririnicolae]

Investigate to which extent models from scikit-learn can fit under the Classifier interface. Some features from the ART API might not be available for this framework. The analysis should cover what those features are. Similarly, if ART can only support some models from scikit-learn, a list of these should be provided.

Depending on the result of the analysis, the list of features to be implemented will be scoped.

[imolloy]

I can help with this one.

[beat-buesser]

I'm interested to help

[ririnicolae]

@beat-buesser @imolloy I think this is quite a big task, so I suggest you share responsibilities. One of you could perform the analysis discussed above. Based on that, you can split the implementation between you. Let me know what you think.

[imolloy]

@ririnicolae Completely agree. We should start with a list of the models we're interested in, e.g., classifiers, regression, etc. Next, there's a question about whether or not the existing sklearn interfaces are going to be sufficient, or if we need to be model-specific. For example, using the fit, predict, predict_proba, transform, fit_transform will be sufficient (implies blackbox attacks) or if we need to dig into any of the models themselves, e.g.,.coef_ and intercept_ for LinearRegression and SVC (along with dual_coef_), and so on.

My initial target would be SVC and LogisticRegression.

[ririnicolae]

@imolloy I think classification is a good place to start, and we should at least be able to cover SVC & LogisticRegression, as you suggested. One of the questions at this point is: will we be able to extract gradients from scikit-learn? These would be vital for white-box attacks (class_gradient and loss_gradient in the Classifier API). But you're right, we can always limit the support to black-box attacks if we can't get gradients.

[imolloy]

Looking at the main modules and classes, this list might get us started. I don't think we'll be able to write a generic gradient function in a whitebox setting, but we can default to a blackbox approximation using something similar to ZOO or NES. We can try to computing gradients for some of the simpler classes and see how it goes.

• [x] sklearn.linear_model.{LinearRegression, LogisticRegression}
• [x] sklearn.tree.{DecisionTreeClassifier}
• [x] sklearn.svm.{SVC}
• [ ] sklearn.naive_bayes.{}
• [ ] sklearn.preprocessing.{}
• [x] sklearn.ensemble.{RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier}
• [ ] sklearn.discriminant_analysis.{LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis}
• [ ] sklearn.decomposition.{NMF, PCA}
• [ ] sklearn.neighbors .{}

[beat-buesser]

Sounds good to me. I have created a new development branch development_sklearn and will start to build there prototypes for sklearn.linear_model.LogisticRegression to explore some of the challenges ahead.

[beat-buesser]

I have pushed a prototype of a classifier for sklearn.linear_model.LogisticRegression to branch development_sklearn in 49a9429d5bfe10c464997f9a6a633cedae87c5e6.

The new notebook sklearn_logistic_regression.ipynb includes examples using the MNIST dataset and art.attacks.projected_gradient_descent.ProjectedGradientDescent.

So far I can see a few TODOs remaining and I'll continue working on them:

• [x] account for class_weights: implemented in bdc1b264b1dd2100853f41139e14de45a2c32ddc
• [x] vectorisation: implemented in 82b4bcbd4291b430516d4731fe28a2e41ce189a0
• [x] implement remaining art.classifier functions: implemented in 6ac03e0efe901d7d07be17f39b0186d26b89f2db, e7e2245223354b635eedb7043a2ce311e02f525b
• [x] targeted attacks: implemented in 26f56b813827db62d65c16254dc9d6c12f0e682d
• [x] add unit tests: implemented in 63c659b8e97aff7fb5e0f42f8fd4e6ce5ec32140

Please let me know what you think.

[beat-buesser]

I have created a new development branch development_sklearn_SVM and started to implement support for Support Vector Machines.

[beat-buesser]

I started working on a classifier for sklearn.svm.SVC. This is a short list of items that I'm implementing:

• [x] gradients for linear kernel
• [ ] gradients for polynomial kernel
• [x] gradients for radial basis function kernel
• [ ] gradients for sigmoid kernel
• [x] account for sklearn.svm.LinearSVC
• [x] account for probability option

[beat-buesser]

The ART classifier for sklearn.svm.SVC has now loss_gradients for linear and rbf kernels. Check out the PGD attack examples on Iris and MNIST datasets in sklearn_svm_svc.ipynb

[beat-buesser]

I have created a new development branch development_decision_trees for the exploration and development on decision tree classifiers from sklearn.tree.DecisionTreeClassifier, sklearn.ensemble.{RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier}, XGBoost, LightGBM and CatBoost and related attacks and defenses.

This is an interesting, very recent article on adversarial examples and robustness for decision trees: https://arxiv.org/abs/1902.10660

[beat-buesser]

Progress with tree-based classifiers:

• [x] sklearn.ensemble.AdaBoostClassifier
• [x] sklearn.ensemble.BaggingClassifier
• [x] sklearn.tree.DecisionTreeClassifier
• [x] sklearn.tree.ExtraTreeClassifier
• [x] sklearn.ensemble.ExtraTreesClassifier
• [x] sklearn.ensemble.GradientBoostingClassifier
• [x] sklearn.ensemble.RandomForestClassifier
• [x] XGBoost
• [x] LightGBM
• [x] CatBoost

[beat-buesser]

All new classifiers and example notebooks have been merged to branch development_sklearn. There development will continue to implement unit tests and improve notebooks.