Added incremental building of ensembles and fixed adaboost

[nitbix]

Adaboost implementation had a bug which resulted in the D vector of dataset weights being incorrect and in some extreme cases leading to non-termination and infinity errors.

Ensembles now can have "steps" where they have a member added on to the existing ensemble. This only works for some ensemble techniques that can be expanded this way.

[nitbix]

I merged this last night with master in my local repo so should be a straightforward diff, although there are lots of commits listed here.

[jeffheaton]

Thank you. I will get this merged in soon!

[jeffheaton]

I am holding on merging this change for now. There are quite a few potentially breaking changes to core Encog classes. Mostly adding the dropout rate the the update weight. This function is called quite a bit in multi-threaded code. Generating a random number for every weight update, even when we have no dropout rate really drags performance. Additionally, the code is creating a new random number generator for each weight update. I like the changes overall, but allocating a new random number generator for every weight update is good for performance.

[nitbix]

Those are valid points. I think some of those are covered in some of my local commits, I'll put together a fix. On 30 Dec 2013 21:34, "Jeff Heaton" notifications@github.com wrote:

I am holding on merging this change for now. There are quite a few potentially breaking changes to core Encog classes. Mostly adding the dropout rate the the update weight. This function is called quite a bit in multi-threaded code. Generating a random number for every weight update, even when we have no dropout rate really drags performance. Additionally, the code is creating a new random number generator for each weight update. I like the changes overall, but allocating a new random number generator for every weight update is good for performance.

— Reply to this email directly or view it on GitHubhttps://github.com/encog/encog-java-core/pull/138#issuecomment-31381154 .

[nitbix]

I just pushed a few changes that address the problems you mentioned. You may also want to ignore the changes to DetermineWorkload.java - I only did them in my repo because I got a small speed gain when using a large cluster, but this may very well not be the case everywhere. If it makes things easy I can do an upstream pull as well, since it's been a while.

[jeffheaton]

Thank you very much for adding this code, and sorry the merge has taken so long. As soon as I finish my current book (August) getting this merged into Encog is my next priority. I am sure I will have some questions.

[nitbix]

No worries Jeff. I've been finding bugs here and there so it's been giving me time to sort it out. I'll do a fresh rebase before you start merging as well to make it easier. And write javadoc.. On 1 Jul 2014 17:27, "Jeff Heaton" notifications@github.com wrote:

Thank you very much for adding this code, and sorry the merge has taken so long. As soon as I finish my current book (August) getting this merged into Encog is my next priority. I am sure I will have some questions.

— Reply to this email directly or view it on GitHub https://github.com/encog/encog-java-core/pull/138#issuecomment-47678108.

[jeffheaton]

Okay that sounds good.

[jeffheaton]

One more question. I am submitting an article to the open source track of the Journal of Machine Learning Research (JRML) for Encog. Your ensemble code is one of the larger blocks of contributed code, so I plan to list you in the acknowledgements (assuming that is okay). Do you have a paper, or similar, that you would like me to cite? I think your code will be very useful to the Encog community once we get it fully integrated, documented, exampled, and hooked into the Workbench. This is what I plan to help with soon.

[nitbix]

It is absolutely okay to list me in the acknowledgements. I have a paper under review which makes use of both dropout and ensembles. Given I can't just assume this gets accepted, you may want to use my MSc thesis and a generic "under review" entry. Bibtex at the bottom for your convenience (first one before acceptance, second one after, third is the dissertation which covers the implementation of ensemble methods). Also if you need to list affiliations, it's "Department of Computer Science and Information Systems, Birkbeck, University of London, Malet Street, London WC1E 7HX ". Let me know if there's anything I can help with!

@INPROCEEDINGS{Mosc1409:Empirical, AUTHOR="Alan Mosca and George Magoulas", TITLE="An Empirical Comparison of Neural Ensembles and Neural Dropout", BOOKTITLE="Submitted to Sixth Computer Science and Electronic Engineering Conference 2014 (CEEC'14)", ADDRESS="Colchester, Essex, United Kingdom", MONTH=sep, YEAR=2014, }

@INPROCEEDINGS{Mosca:Empirical, AUTHOR="Alan Mosca and George Magoulas", TITLE="An Empirical Comparison of Neural Ensembles and Neural Dropout", BOOKTITLE="Sixth Computer Science and Electronic Engineering Conference 2014 (CEEC'14)", ADDRESS="Colchester, Essex, United Kingdom", MONTH=sep, YEAR=2014, }

@mastersthesis{Mosca:mscdissertation, author = {A Mosca}, title = {Extending Encog: A study on classifier ensemble techniques}, school = {Birkbeck, University of London}, year = {2012} }

On 2 July 2014 15:21, Jeff Heaton notifications@github.com wrote:

One more question. I am submitting an article to the open source track of the Journal of Machine Learning Research (JRML) for Encog. Your ensemble code is one of the larger blocks of contributed code, so I plan to list you in the acknowledgements (assuming that is okay). Do you have a paper, or similar, that you would like me to cite? I think your code will be very useful to the Encog community once we get it fully integrated, documented, exampled, and hooked into the Workbench. This is what I plan to help with soon.

— Reply to this email directly or view it on GitHub https://github.com/encog/encog-java-core/pull/138#issuecomment-47781887.

Alan

[jeffheaton]

Thank you, the bibtex is useful. I added a cite to your master's thesis. I also list you (with affiliation) in the acknowledgments section. I've not worked with dropout or ensemble for neural networks yet, and mostly understand them in concept. I am looking forward to adding this to Encog, and learning more myself. Thank you for you willingness to add this to Encog. Best of luck with your paper, and I will let you know how the Encog paper with JMLR turns out.

[nitbix]

Thank you, that's very kind. I have been using Encog for the experiments comparing Ensembles and Dropout that are in this paper, so it seems only fair to contribute the code back. I have been working on a tutorial paper on Ensembles for ANN, which uses one of the comparisons as an example. It's not finished, but once it's ready I'll send across a copy. I'll also send you a copy of the comparisons paper off-list as soon as I get an ok from supervisor who co-authored it.

On 5 July 2014 18:35, Jeff Heaton notifications@github.com wrote:

Thank you, the bibtex is useful. I added a cite to your master's thesis. I also list you (with affiliation) in the acknowledgments section. I've not worked with dropout or ensemble for neural networks yet, and mostly understand them in concept. I am looking forward to adding this to Encog, and learning more myself. Thank you for you willingness to add this to Encog. Best of luck with your paper, and I will let you know how the Encog paper with JMLR turns out.

— Reply to this email directly or view it on GitHub https://github.com/encog/encog-java-core/pull/138#issuecomment-48091959.

Alan

[nitbix]

Hi Jeff, I noticed that some of the new Ensemble classes have appeared in the main repo. I don't know if you are midway through the merge or have managed to get everything to work decently, but please do let me know if there's anything I can do to help.

[jeffheaton]

I did add a few classes from an earlier attempt to merge it. I hope to release Encog 3.4 before the end of the year. It would be great to get your code added to 3.4. What would be the best way? Have you added/fixed anything beyond what is currently in this pull request?

If it would not be too much trouble for you to update to the current 3.3 code and then send another pull request, I would be glad to review any compatibility issues where your code intersects Encog and adjust as needed.

[nitbix]

That sounds like a good idea. I ended up making the pull request from HEAD so it contains all my changes, but perhaps I can fix it so that it merges. I'll do that and send another pull request. Thanks! On 14 Oct 2014 18:46, "Jeff Heaton" notifications@github.com wrote:

I did add a few classes from an earlier attempt to merge it. I hope to release Encog 3.4 before the end of the year. It would be great to get your code added to 3.4. What would be the best way? Have you added/fixed anything beyond what is currently in this pull request?

If it would not be too much trouble for you to update to the current 3.3 code and then send another pull request, I would be glad to review any compatibility issues where your code intersects Encog and adjust as needed.

— Reply to this email directly or view it on GitHub https://github.com/encog/encog-java-core/pull/138#issuecomment-59086521.

[jeffheaton]

I finally merged this in, very sorry about the long delay on this one. I've actually gotten into some of the technology you've added, so this is very interesting for me to go through this. I am going to experiment with my existing Encog code and make sure everything works as I expect, and will make any changes as needed.

[nitbix]

Thanks, do let me know if anything is unclear or looks wrong!

On 7 July 2015 at 18:29, Jeff Heaton notifications@github.com wrote:

I finally merged this in, very sorry about the long delay on this one. I've actually gotten into some of the technology you've added, so this is very interesting for me to go through this. I am going to experiment with my existing Encog code and make sure everything works as I expect, and will make any changes as needed.

— Reply to this email directly or view it on GitHub https://github.com/encog/encog-java-core/pull/138#issuecomment-119276410 .

Alan

[jeffheaton]

Just one small thing I noticed, the FlatNetwork does have a default constructor that is used sometimes with serialization. It does not call init, so the dropout percentage variable was not getting set and causing NullPointerException's later on. The following very simple fix resolves that:

https://github.com/encog/encog-java-core/commit/03613385836192b5ef91b1dd5f7037d852df4fc7

Otherwise all of the previous code seems to be functioning as I would expect!

Thanks again! I am sure I will have more questions as I go through it.

[jeffheaton]

FYI, the paper with my citation to your thesis for this has been published. http://jmlr.org/papers/volume16/heaton15a/heaton15a.pdf