[WIP] Staring a new PR with building on top of sklearn.

[adam2392]

PR Description

Start of PR that aims to:

• clean up code to build upon sklearn code cleanly
• optimize performance of CySPORF
• run validation tests of CySPORF, sklearn RF master branch and RF (via this code) to validate that this code does not affect traditional RF performance

Summary

Summary of Python level API:

• New Forest classifier and regressor class for SPORF
• Slightly modify BaseDecisionTree to allow inheritence for BaseObliqueDecisionTree, which is used in SPORF . Don't need to - rewrite fit() and a bunch of other functions, just need to modify the setting of a splitter, tree and tree-builder.
• [x] TODO: need to incorporate the latest sklearn's Criterion which currently will break our code. Prolly just a few line changes here and there.

Summary of Cython level:

• _criterion -> stays the same (no changes)
• _utils -> include safe realloc for any Oblique data structures. Otw no changes
• _tree -> keep as is (this is for traidional RF)
• _splitter -> keep as is (this is for traditional RF)
• add _oblique_tree and _oblique_splitter cython files for the tree, tree-builder and splitter for SPORF

To test the sklearn's RF code: we will copy the sklearn/ensemble files which include native RF and no need to modify. Run on oblique_forests.RandomForestClassifier and sklearn.RandomForestClassifier and literally they should be identical cuz the code at Python level and Cython won't change.

^ This may not be the case if say sklearn ppl want us to incorporate the Oblique Cython changes into the existing _tree and _splitter files, but imo that doesn't make semantic sense and would complicate the existing codebase imo, rather then just using inheritence to enable a new API, which is what we're doing now.

Merge checklist

Maintainer, please confirm the following before merging:

• [ ] All comments resolved
• [ ] This is not your own PR
• [ ] All CIs are happy
• [ ] PR title starts with [MRG]
• [ ] whats_new.rst is updated
• [ ] PR description includes phrase "closes <#issue-number>"

[adam2392]

One remaining challenge is implementation of pickling via the __getstate__ and __setstate__ functions, which needs some handling of the proj_vecs pointer pointer array.

This will enable cysporf to be passed be pickled when doing something like cross_val_score.

Note that joblib works when "building" forests. Not sure what makes one work but not the other.

[adam2392]

Added in the Criterion files that sklearn v0.24.2 uses.

[adam2392]

Added the ensemble set of files from v0.24.2 to verify that our modifications of the Cython level files under tree/ do not hurt the RandomForest Classifier/Regressor.

[adam2392]

Good call on the refactoring. Do we need to keep the gradient boosting-related files under ensemble? They seem unrelated

Nah we can prolly get rid of those. I just copied everything in :p. But it seems that at least ensemble/_forest.py does not rely on those files, so we're okay.