Automatic grouped analysis of committee members

mtewes commented 7 years ago

Things I have in mind:

[x] an automatic checkplot (controlled by the same option as the other training checkplots) that visualizes the cost function (evolution ?) of the committee members
[x] a summary (logger.info ?) about the committee (a kind of table giving how well and how long each member did / took)
[ ] a bit more verbosity (logger.info) about the "reduction function" for predictions (this is in megalut, not tenbilac). Currently, when it runs, the log doesn't say why nothing happens for ages.

mtewes commented 7 years ago

Starting this now, in the dedicated branch #15

mtewes commented 7 years ago

After discussion with @kuntzer , this branch will become an alternative implementation of committees, acting as a wrapper for the whole tenbilac process (setting up, norming, training, analysing, and then predicting, denorming), and relying on YAML config file(s). To keep the code simple (and explicitly avoid confusing inheritances and redelarations) the wrapper will "hard code" committees (instead of first redesigning committees, and then a config-file-wrapper around all this).

Aims are:

make it trivial to compare, trace, and share tenbilac settings (-> config files which can also be updated by code)
committee members are just directories and config files, they can be moved around.
MegaLUT's tenbilacwrapper should become empty again (really just setup, train, predict)
predicting with megalut should no longer require a TenbilacParams object (See issue https://github.com/megalut/megalut/issues/125). Just a catalog + MLParams saying what columns to use, and the path to a Tenbilac config file. This config file will tell everything, including the way to the Net pickles, using only relative paths.
when predicting on acutal data, the merging of the results from commitee members should also be done "internally" here in tenbilac (currently this happens in megalut.learn.ml), again controlled by the config file.
Avoid duplications (of Nets, Training pickles Committee pickles)

mtewes commented 7 years ago

@kuntzer I openend up a PR about this: #29 , even if it's far from "done". But If you have time, feel free to checkout branch 15 and run the quick demo in ./demo/com/. I'm offline for the next hour or so (maybe even until 16:30 ?), but any comments are of course welcome (or we just skype about it). ConfigParser is a bit weird. I'm still confused how to nicely pass a Section of a config to (e.g.) a particular method, given that one has to hard-code getint, getfloat, getboolean when reading from the config 😞

mtewes commented 7 years ago

Back to this issue. Instead of YAML we use ConfigParser, and multiprocessing works as well (same as with committee.py). Now focusing on the actually "new" stuff from this branch: summary of training results, and committee-predictions.

mtewes commented 7 years ago

Done with first a first version of this summary error evolution plot. summaryerrevo-1 Not super-smart, but flexible and the code is short.

Will go on with writing "predict"...

mtewes commented 7 years ago

Predict is implemented, takeover as well... summaryerrevo

mtewes commented 7 years ago

Closing. No need for verbosity on the reduction, as this happens in tenbilac and is very fast now.

mtewes / tenbilac

Automatic grouped analysis of committee members #15