Command-line capability of running cross-validation

[dahhei]

Feature request for being able to run cross-validation from the command line. When a project has many videos for cross-val, the all k-fold cross-val takes a long time. 😕

[SkepticRaven]

It's not documented, but it looks like https://github.com/KumarLabJax/JABS-behavior-classifier/blob/main/stats.py does exactly this, when you pass a -k greater than the number of possible k-folds. It only operates on exported classifiers in its current state.

There's a handful of upgrades to it that would make it more useful:

• Allow either project folders or saved classifiers.
• Add an easier way to just say "do all k-fold".
• Allow easy access to window sizes and feature subsets/setup routines for automated hyperparameter optimization
  • Requires projects, since features would need to be re-calculated
  • (Longer term) Add in Nicks efforts for ML hyperparameters (https://github.com/KumarLabJax/JABS-behavior-classifier/pull/11/commits/f664540488488ddbc1ce8da7f7e0847f3a489aca)