Closed tjrocha closed 5 years ago
I think this could be useful, particularly for testing the efficacy of the feature selection algorithm(s). Would you do it automatically for any predictor set less than an upper limit, or allow the user to select it as an option and warn them if they are going to overwhelm the computer?
Thanks for the feedback. I was thinking of making it a selectable option available in the Regression Tab right above the Forward and Backwards Selection methods. Yeah, warning the user would be a good idea based on whether the number of equations to evaluate exceeds an arbitrary large number.
The issue would be run-time since the program would have to evaluate all the models but in terms of memory, we can have it only store the top X models so if new better performing models are evaluated, they can just knock-off a worse performing model so we don't run out of memory and overwhelm the computer.
Fully working and tested implementation with 8112f27.
Just spit-balling here... Would it be useful to have a Brute Force Feature Selection method incorporated into the software? For some relatively simple models (#Predictors = 15), we can just brute force evaluate all the possible combinations of predictors (#Combinations=32,767) and have the program report the top performing models. Benefits to this become more apparent if there are fewer predictors since the brute force number of equations are of the form (2^n)-1 with n=#Predictors.
On a related note, I noticed that the existing Feature Selection algorithm evaluates the same model multiple times especially if it is not selected/stored in the list of viable regression models. There might be some performance gains to maybe storing in memory just the salient metrics (Predictor IDs & Selected Metric) for every model run and referring to this in-memory object so the algorithm doesn't evaluate the same model multiple times.