Automatic selection of committee members

[mtewes]

One could have an option "usebestfrac = 0.5" meaning that only the best half of the members get used to compute predictions.

[kuntzer]

agreed! We also could give usebest = 4

[mtewes]

Yes, good point. How do we define "best"? A Net does not know how good it is. A Training does, so we should keep this info (once defined) when minimizing. Or we could end a training with writing a "configuration file" listing the members and their performance. And reading this file (if found) to identify which members should be used. I kind of like this, even if it's sophisticated.

[kuntzer]

I like this as well. But I would write it a bit more often than once per training. It's a bit annoying to do, but it makes sure that we still can predict while training (useful on unstable systems)

[mtewes]

Yes, it would also be good to have the summary plot more often.... but each member runs on a different thread :hurtrealbad:

[kuntzer]

ahah, that's a nice smiley. Either we evaluate asynchronously the performance (which has issues of how we can store the performance & multiprocessing lock problems) or we wait at given points to evaluate all nets at the same time (which brings up issues of run time).

[mtewes]

Will try to implement the simple variant "usebest = 4": trainings know how good they are, and a selection of committee members is done when predicting based on these "saved" goodness on the validation data.

It's a key feature when training 200 committees for GREAT3.

Should be easy to develop something more sophisticated when needed.

[mtewes]

Will work on this issue now, I need it also when training weights.

[mtewes]

Done, closing. We might need something smarter in future (sigma-clipping bad members), but for now this is it.