search

mlpack / benchmarks

Machine Learning Benchmark Scripts
101 stars 49 forks source link

Enhancements to benchmarking #146

Open adiwajshing opened 4 years ago

adiwajshing commented 4 years ago

Hi,

I was thinking of adding some features & simplifying the benchmarking tool into a single line usage command line interface.

For example:

  1. ‘mlpack-benchmark -ann -mlpack -github=#1123123’
  2. ‘mlpack-benchmark -ann -shogun -github=#522323’
  3. ‘mlpack-benchmark -ann -mlpack'

The following should be the parameters to this system:

  1. The algorithm (-ann, -kmeans, -cnn)
  2. The ML library (-mlpack, -shogun, -scikit)
  3. GitHub commit (optional)
  4. Local path to uncommitted library (-local=/some/path)
  5. Which data sets to train on (-datasets=wine,iris)

The output should be the time it took to train on each data set, error rate, more specific output on the algorithm itself (MSE, avg. time per epoch etc.)

This enhancement should also allow users to specify which commit of the library they wanna run the benchmark test on. Alternatively, they could specify a local commit to bench mark, to easily test uncommitted changes. Moreover, this program will automatically download & build the source of the library if required.

rcurtin commented 4 years ago

This seems like it would be an extremely significant refactoring. I don't have a problem with it (it sounds like a cool idea to me!), I just want to point out that it seems likely to take many weeks of hard work to make that work correctly.

zoq commented 4 years ago

Agreed nice idea, I think some parts are already there, but they have to be put together in the right way.

adiwajshing commented 4 years ago

@rcurtin @zoq I'm willing to put the work in, it should actually be quite exciting! I had trouble benchmarking everything when I was working on this and so, figured a uniform & simply way to benchmark ML algorithms would go a long way in helping everybody who wants to test out new changes etc.

Also, I had proposed this in my GSOC application, I hope it's okay if I start this now?

rcurtin commented 4 years ago

Sure, feel free, just be aware that it might be a while until any of us are able to review it. :)

adiwajshing commented 4 years ago

No worries, it'll take a while to finish anyway!