Comparison between accuracies of SPORF and sklearn RFClassifier under default parameter and optimized hyperparameter conditions

[sju777]

Reference Issues/PRs

• Fixes #3 in terms of comparing both classifiers under default parameter and optimized hyperparameter conditions

What does this implement/fix? Explain your changes.

• Gather accuracy and runtime data in benchmarking for SPORF and sklearn RF classifier using the OpenML CC-18 and 100 suites
• Generate figures to compare accuracy and runtime data between SPORF and RF under default parameter and optimized hyperparameter conditions

Any other comments?

• Because a lot of the datasets took an incredibly long time to run (12+ hours), even on MARCC, I chose 12 smaller datasets to benchmark using optimized hyperparameters. That said, from the comparisons I was able to perform, it is fairly clear that SPORF performs more accurately than RF (SPORF performed better for 11 out of 12 datasets).

PR is complete, but any and all feedback is welcome!

[j1c]

Can you show me the marcc bash file? The clusters should never time out

[sju777]

Can you show me the marcc bash file? The clusters should never time out

I just uploaded my marcc bash file and the .py file I was running with it!

[j1c]

Idk how far along you are, but I would change the option to below. This will run at most 3 days.

#SBATCH --partition=lrgmem
#SBATCH --nodes=1
# number of tasks (processes) per node
#SBATCH --ntasks-per-node=48
#SBATCH --mem=500G

[sju777]

Idk how far along you are, but I would change the option to below. This will run at most 3 days.

#SBATCH --partition=lrgmem
#SBATCH --nodes=1
# number of tasks (processes) per node
#SBATCH --ntasks-per-node=48
#SBATCH --mem=500G

I am currently trying to get the last 5 datasets of the OpenML CC18 set, but if it times out again, I will use the options you gave me.