Grid Analyzer!

[tkornuta-ibm]

Finally, working mip-grid-analyzer is here!

This required to solve several issues and introduce many changes in the whole experimental pipeline, starting from:

• rethinking what we really want to show in the final csv file - fixes #66
• investigating the content of statistics and rethinking the desired behavior of online/offline trainers (fixes #64 )
• store termination cause (fixes #67 ) along with testing and validation statistics (fixes #74 ) in model checkpoint
• fix formatting issue in grid-analyzer(fixes #77 )
• I have also introduced fixes to all grid-workers-cpu that enables to distribute computations into several processes on CPU on MacOS (fixes #52 ) (WARNING! Didn't tested that on Ubuntu CPUs nor GPUs, should work, but needs testing... - creating new issues #82 and #83 )

In order to test the whole pipeline, I have also prepared a config for grid_trainer with LeNet5 and SimpleCNN model trained on MNIST (fixes #51 )

The use case is as follows:

• run grid-trainer on the configuration file (2 models, 1 problem, 5 repetitions) 'mip-grid-trainer-cpu --c configs/vision/grid_trainer_mnist.yaml'

• run grid-tested on the newly created directory with containing 10 different models 'mip-grid-tester-cpu --e --expdir ./experiments_20181109_131153'

• run grid-analyzer to aggregate all the results in a single csv file 'mip-grid-analyzer --e --expdir ./experiments_20181109_131153'

Attaching the resulting file:

20181109_134802_grid_analysis.xlsx

(What I have described above will in fact become a part of "reproducible research: MNIST Image Classification with LeNet-5 - issue #12 )

Besides, I've made plenty of minor changes, like:

• added Ctrl+C handling to grid workers (fixes #63 )
• changed --o to --e (fixes #65 )
• cleaned up some of the arguments of grid workers (partially #25 )
• adds option to select basic trainer from both command line and config of gridtrainer* (fixes #62)

[tkornuta-ibm]

This pull request introduces 3 alerts when merging 1ae12d3e5aed6b4e850f4fea549f5c93c627be33 into d8c20f1e2a50e21c4a80ce8fecdb7e480395b4d3 - view on LGTM.com

new alerts:

• 3 for Unused import

---

Comment posted by LGTM.com

[vmarois]

It looks good! I haven't add the time to test it so will do so on Monday.

This PR adds a dependency over psutil.. Is there no way around it?

[tkornuta-ibm]

psutil

Sadly not, please review the solutions that I have analyzed when fixing issue #52