Waikato / wekaDeeplearning4j

Weka package for the Deeplearning4j java library
https://deeplearning.cms.waikato.ac.nz/
GNU General Public License v3.0
185 stars 202 forks source link

MultipleEpochsIterator not working right with ShufflingDataSetIterator #7

Closed christopher-beckham closed 8 years ago

christopher-beckham commented 8 years ago

I am not sure if such a thing exists somewhere, but I had to write my own iterator that can return mini-batches of a dataset, and (optionally) shuffle it as well. It seems to not work properly when used in conjunction with the multiple epoch iterator that comes with DL4J. For example, if ShufflingDataSetIterator is used on its own on iris.arff and we specify a batch size of 50, we expect 3 mini-batches (since there are 150 instances total). However, if we wrap this in MultipleEpochsIterator and specify 10 epochs, we get 28 mini-batches, instead of the proper 10*3 = 30 mini-batches.

I have reproduced this phenomena in the file IrisTest.java

christopher-beckham commented 8 years ago

For now I have removed the use of the multiple epochs iterator, and just have a for loop in the buildClassifier script that does the same thing.